This document discusses key concepts in language assessment including validity, reliability, and feasibility. It provides definitions and examples of different types of validity including construct, content, criterion-related, and face validity. Reliability is discussed in terms of test-retest, alternate forms, and split-half methods. The document also covers types of language assessment such as proficiency tests, achievement tests, and diagnostic tests. Specific techniques for assessing writing, speaking, reading, listening, grammar, and vocabulary are outlined. Guidelines are provided for developing valid and reliable language tests.
This document discusses oral ability testing for language proficiency. It outlines various considerations for setting tasks, sample criteria, test formats, elicitation techniques, scoring procedures, and training for test administrators. The key points covered are setting clear objectives for testing comprehension and production, selecting appropriate content and text types for tasks, using multiple formats and testers to obtain a representative sample, carefully planning interviews and ensuring candidate comfort, and training scorers to reliably evaluate performance based on specified criteria.
1) The document discusses various test techniques for measuring language abilities, including multiple choice, cloze, C-test, and dictation.
2) Multiple choice tests recognition knowledge but has reliable scoring while cloze and C-tests measure reading ability but are more difficult to score.
3) No single technique is sufficient and combining scores from different techniques that measure separate abilities can provide a more overall assessment of language proficiency.
This document discusses English for Specific Purposes and the process of needs analysis to identify performance requirements and gaps in what is required versus present abilities. It outlines the steps in needs analysis, including deciding the objects of analysis, selecting data collection methods/instruments, collecting and analyzing data, and using the results to design courses and syllabi. Some methods of needs analysis mentioned are entry tests, class observations, questionnaires, interviews, and learner diaries. The document also discusses pedagogical, sociological, linguistic, and psychological considerations in needs analysis. It concludes by asking students to bring information on the group they will work with to start designing a needs analysis instrument.
The document summarizes the educational method known as Suggestopedia, which was developed in the 1980s-1990s. It uses relaxation, music, and suggestion to optimize language learning. Key elements include a positive environment with pictures, color, music, and varied active methods like songs and games. Classes follow four stages - presentation, active concert, passive review, and practice. The goal is to teach languages 3 times faster than conventional methods by relaxing students and presenting material in an engaging way while they are in a receptive state.
This document discusses assessment in the CLIL (Content and Language Integrated Learning) context. It addresses three main issues: 1) Whether to assess content, language, or both, with content generally prioritized. 2) The importance of clear learning objectives. 3) Using a variety of assessment methods and instruments, both formal and informal. These include exams, tasks, rubrics, portfolios, and self-assessment to minimize language impact and effectively evaluate content knowledge and language skills. Formal assessments should have low linguistic risk while informal and formative assessments provide ongoing feedback to improve learning.
This document discusses testing oral ability. It outlines several key considerations for developing valid and reliable oral tests, including using representative tasks that sample a variety of language functions, ensuring tasks elicit targeted language samples, and using different testing formats like interviews, paired interactions, and responses to audio/video prompts. It also emphasizes the importance of carefully planning and structuring tests, providing candidates multiple opportunities to demonstrate their skills, selecting appropriate rating scales and trainings raters to score tests validly and reliably.
Five Fun Activities to Build Listening Skillsallisg43
Can listening activities be fun and motivating? These slides look at listening in the EFL classroom and outline five fun and easy-to-use activities to help EFL learners build listening skills in an enjoyable and exciting way. Material from the e-future texts Listen Up and Listen Up Plus are used in the slides.
These slides are from a presentation delivered at KOTESOL in Seoul on October 12th, 2013.
This document discusses oral ability testing for language proficiency. It outlines various considerations for setting tasks, sample criteria, test formats, elicitation techniques, scoring procedures, and training for test administrators. The key points covered are setting clear objectives for testing comprehension and production, selecting appropriate content and text types for tasks, using multiple formats and testers to obtain a representative sample, carefully planning interviews and ensuring candidate comfort, and training scorers to reliably evaluate performance based on specified criteria.
1) The document discusses various test techniques for measuring language abilities, including multiple choice, cloze, C-test, and dictation.
2) Multiple choice tests recognition knowledge but has reliable scoring while cloze and C-tests measure reading ability but are more difficult to score.
3) No single technique is sufficient and combining scores from different techniques that measure separate abilities can provide a more overall assessment of language proficiency.
This document discusses English for Specific Purposes and the process of needs analysis to identify performance requirements and gaps in what is required versus present abilities. It outlines the steps in needs analysis, including deciding the objects of analysis, selecting data collection methods/instruments, collecting and analyzing data, and using the results to design courses and syllabi. Some methods of needs analysis mentioned are entry tests, class observations, questionnaires, interviews, and learner diaries. The document also discusses pedagogical, sociological, linguistic, and psychological considerations in needs analysis. It concludes by asking students to bring information on the group they will work with to start designing a needs analysis instrument.
The document summarizes the educational method known as Suggestopedia, which was developed in the 1980s-1990s. It uses relaxation, music, and suggestion to optimize language learning. Key elements include a positive environment with pictures, color, music, and varied active methods like songs and games. Classes follow four stages - presentation, active concert, passive review, and practice. The goal is to teach languages 3 times faster than conventional methods by relaxing students and presenting material in an engaging way while they are in a receptive state.
This document discusses assessment in the CLIL (Content and Language Integrated Learning) context. It addresses three main issues: 1) Whether to assess content, language, or both, with content generally prioritized. 2) The importance of clear learning objectives. 3) Using a variety of assessment methods and instruments, both formal and informal. These include exams, tasks, rubrics, portfolios, and self-assessment to minimize language impact and effectively evaluate content knowledge and language skills. Formal assessments should have low linguistic risk while informal and formative assessments provide ongoing feedback to improve learning.
This document discusses testing oral ability. It outlines several key considerations for developing valid and reliable oral tests, including using representative tasks that sample a variety of language functions, ensuring tasks elicit targeted language samples, and using different testing formats like interviews, paired interactions, and responses to audio/video prompts. It also emphasizes the importance of carefully planning and structuring tests, providing candidates multiple opportunities to demonstrate their skills, selecting appropriate rating scales and trainings raters to score tests validly and reliably.
Five Fun Activities to Build Listening Skillsallisg43
Can listening activities be fun and motivating? These slides look at listening in the EFL classroom and outline five fun and easy-to-use activities to help EFL learners build listening skills in an enjoyable and exciting way. Material from the e-future texts Listen Up and Listen Up Plus are used in the slides.
These slides are from a presentation delivered at KOTESOL in Seoul on October 12th, 2013.
This document discusses testing oral ability through oral exams. It outlines appropriate tasks to test oral ability, including operations like expressing, narrating, and eliciting. It discusses criterial levels of performance in terms of appropriacy, accuracy, range, size, accent, grammar, vocabulary, and comprehension. Possible test formats are also outlined, such as interviews, interactions between candidates, and responses to recordings. Techniques for eliciting behaviors from candidates and obtaining valid and reliable scoring are also discussed.
This document discusses integrating the four language skills of listening, reading, speaking and writing. It defines the integrated skills approach as interweaving the skills for optimal communication, unlike the traditional segregated approach. It describes two basic models of integration - simple integration within the same medium, and complex integration through a series of linked activities. Five models for skills integration are then outlined: content-based instruction, task-based language teaching, theme-based teaching, experiential learning, and using episodes. The document concludes that the integrated skills approach exposes learners to authentic language use and promotes learning real content over isolated language forms.
Introduction to Language Assessment by BrownEFL Learning
This document provides an overview of assessment for EFL learners. It defines assessment as an ongoing process to ensure course objectives are met, noting that a test is one form of assessment. Informal assessments include unplanned feedback, while formal assessments systematically appraise student achievement. Various traditional and alternative assessment types are described, including their purposes and characteristics. The principles of practicality, reliability, validity, authenticity, and washback in language assessment are also outlined.
Dokumen tersebut membahas tentang hakikat dan komponen keterampilan berbicara. Secara garis besar, dokumen menjelaskan bahwa berbicara adalah kemampuan mengubah pikiran menjadi bunyi bahasa yang bermakna, dan terdiri dari penggunaan bahasa lisan, penguasaan isi pembicaraan, serta teknik dan penampilan berbicara. Dokumen juga membahas tentang tes berbicara, jenis tes berbicara, serta te
This document discusses techniques for testing overall language ability, including cloze procedures, dictation, and tests for young language learners. Cloze procedures aim to assess abilities beyond just the immediate context by removing words from a passage. Dictation can test spelling, punctuation, and listening ability but is time-consuming to score. When testing young learners, tests should be brief, varied, and involve visual elements like pictures to match their short attention spans and developing cognitive skills.
This document discusses factors that affect language test scores and reliability. It defines reliability as the proportion of observed score variance due to true score variance. Methods for estimating reliability include internal consistency, stability over time, and equivalence of alternate forms. Internal consistency examines consistency of performance across parts of a test and can be estimated using split-half reliability, which treats halves of a test as parallel forms.
The document discusses developing materials for language courses. It defines developing materials as the process teachers use to create lessons and units to meet course goals and objectives. Teachers must consider factors like course goals, their teaching philosophy, and how students learn when developing materials. Well-designed activities should engage students, build skills and confidence, integrate different skills, and use authentic texts when possible. The document provides examples of materials developed by teachers for courses on topics like family, writing, and relationships. It emphasizes that developing effective materials depends on understanding teachers' beliefs and students' needs in the specific context.
The document discusses different types of language tests, including objective tests, subjective tests, direct tests, indirect tests, discrete-point tests, and integrative tests. It also discusses the differences between norm-referenced tests and criterion-referenced tests. Norm-referenced tests measure students against each other, while criterion-referenced tests measure students against an absolute standard. The key differences are that norm-referenced tests have a broader content focus, aim to discriminate between high and low achievers, and report scores in percentiles compared to peers. Criterion-referenced tests have a specific content domain, measure whether students achieve specific skills, and report scores as a percentage of skills mastered.
This document discusses teaching listening skills in English as a second language. It begins by stating that the objective of classroom listening is to prepare students for real-life listening situations. It then lists common features of real-life listening situations, such as informal language, noise, redundancy, and ongoing listener response.
The document notes that classroom listening does not replicate real-life listening. It recommends basing listening activities on simulated real-life situations to motivate students. Guidelines are provided for listening texts, such as using informal speech, and for listening tasks, such as providing expectations and an ongoing purpose. Finally, examples are given of different types of listening activities and how they can be adapted.
Testing writing (for Language Teachers)Wenlie Jean
The document discusses the key considerations for properly testing writing ability. It identifies four main problems in testing: 1) using representative tasks, 2) eliciting valid writing samples, 3) ensuring scores are valid and reliable, and 4) providing feedback. For each, it outlines various factors that test designers should take into account such as specifying all content domains, including a representative task sample, restricting candidates, using appropriate scoring scales, and calibrating scorers. The goal is to develop writing tests that accurately measure students' abilities.
This document summarizes key aspects of curriculum design approaches from chapters 9 and 10 of the book "Language Curriculum Design" by I.S.P Nation and John Macalister. It discusses three common approaches to the curriculum design process: the waterfall model, focused opportunistic approach, and layers of necessity model. It also covers negotiated syllabuses, where teachers work with learners to make joint decisions about curriculum design elements. Requirements for implementing a negotiated syllabus include establishing negotiation procedures, planning course content and activities, setting learning goals, and evaluating outcomes.
The document discusses assessing writing skills. It describes different types of writing like academic, job-related, and personal writing. It outlines micro skills like imitative and intensive writing, and macro skills like responsive and extensive writing. For micro skills, it provides examples of assessment tasks for imitative writing like spelling tests and dictation. For intensive writing, it discusses tasks like rewriting sentences and transforming grammar. For macro skills, it discusses designing assessment tasks for responsive and extensive writing like guided questions, paragraph construction, and scoring methods.
Language testing is the practice of evaluating an individual's proficiency in using a particular language. There are two main types of assessment: formative assessment which checks student progress, and summative assessment which measures achievement at the end of a term. There are five common types of language tests: proficiency tests which measure overall ability, achievement tests related to course content, diagnostic tests which identify strengths and weaknesses, placement tests for assigning students to class levels, and direct/indirect tests. The effect of testing on teaching is known as backwash, which can be harmful if not aligned with course objectives, or beneficial if tests influence instructional changes.
This document provides an overview of linguistic approaches to analyzing ideology between 1979 and 2010. It discusses three main approaches: Critical Linguistics (CL), Critical Discourse Analysis (CDA), and Critical Stylistics (CS). CL, introduced in 1979, used linguistic analysis and Systemic Functional Grammar to analyze ideology in texts. CDA, developed in 1989, expanded CL to include social and institutional contexts. Between 1989-2010, various CDA theories emerged. CS, introduced in 2010, returned linguistic analysis to the forefront by drawing on CL and CDA. The document reviews developments across these three decades in applying linguistic theories to analyze textual ideologies.
This document discusses assessment in language learning. It defines assessment as collecting information about students' language development through various methods such as tests, portfolios, and observations. This information is then analyzed and used to make pedagogical decisions. For assessment to be effective, it should be valid, reliable, and feasible. Validity means the assessment accurately measures proficiency. Reliability means a student would achieve similar results on multiple attempts. Feasibility means the assessment is practical to implement. The document also discusses formative assessment, self-assessment, and the importance of feedback in the learning process.
This document discusses different types of language tests and testing, including proficiency tests, achievement tests, diagnostic tests, placement tests, direct and indirect testing, discrete point and integrative testing, norm-referenced and criterion-referenced testing, objective and subjective testing, and computer adaptive testing. It provides details on the purpose and characteristics of each type of test.
This document distinguishes between language systems and language skills. It discusses the key components of language systems, including phonology, lexis, grammar, morphology, syntax, and functional and situational aspects. It also outlines the four main language skills - listening, speaking, reading, and writing - and how they relate to each other as receptive or productive skills involving spoken or written communication. The document emphasizes that an integrated approach to teaching language systems and skills is important for effective language learning and communication.
This document discusses key concepts in language assessment including validity, reliability, and feasibility. It defines validity as the accuracy of a test in measuring the intended proficiency. There are different types of validity including content, criterion-related, and construct validity. Reliability refers to a test producing consistent results, which can be measured using methods like test-retest. Feasibility means a test is practical to administer. The document also discusses types of language tests, how to improve validity and reliability, and item analysis. Chapters from a book on language testing techniques are assigned for discussion.
The Common European Framework of Reference (CEFR) is a standard developed by the Council of Europe to describe language ability. It introduces six common reference levels (A1 to C2) to standardize language education across Europe. The CEFR provides clear definitions of what language learners can do at each level to facilitate cooperation in language education.
This document discusses testing oral ability through oral exams. It outlines appropriate tasks to test oral ability, including operations like expressing, narrating, and eliciting. It discusses criterial levels of performance in terms of appropriacy, accuracy, range, size, accent, grammar, vocabulary, and comprehension. Possible test formats are also outlined, such as interviews, interactions between candidates, and responses to recordings. Techniques for eliciting behaviors from candidates and obtaining valid and reliable scoring are also discussed.
This document discusses integrating the four language skills of listening, reading, speaking and writing. It defines the integrated skills approach as interweaving the skills for optimal communication, unlike the traditional segregated approach. It describes two basic models of integration - simple integration within the same medium, and complex integration through a series of linked activities. Five models for skills integration are then outlined: content-based instruction, task-based language teaching, theme-based teaching, experiential learning, and using episodes. The document concludes that the integrated skills approach exposes learners to authentic language use and promotes learning real content over isolated language forms.
Introduction to Language Assessment by BrownEFL Learning
This document provides an overview of assessment for EFL learners. It defines assessment as an ongoing process to ensure course objectives are met, noting that a test is one form of assessment. Informal assessments include unplanned feedback, while formal assessments systematically appraise student achievement. Various traditional and alternative assessment types are described, including their purposes and characteristics. The principles of practicality, reliability, validity, authenticity, and washback in language assessment are also outlined.
Dokumen tersebut membahas tentang hakikat dan komponen keterampilan berbicara. Secara garis besar, dokumen menjelaskan bahwa berbicara adalah kemampuan mengubah pikiran menjadi bunyi bahasa yang bermakna, dan terdiri dari penggunaan bahasa lisan, penguasaan isi pembicaraan, serta teknik dan penampilan berbicara. Dokumen juga membahas tentang tes berbicara, jenis tes berbicara, serta te
This document discusses techniques for testing overall language ability, including cloze procedures, dictation, and tests for young language learners. Cloze procedures aim to assess abilities beyond just the immediate context by removing words from a passage. Dictation can test spelling, punctuation, and listening ability but is time-consuming to score. When testing young learners, tests should be brief, varied, and involve visual elements like pictures to match their short attention spans and developing cognitive skills.
This document discusses factors that affect language test scores and reliability. It defines reliability as the proportion of observed score variance due to true score variance. Methods for estimating reliability include internal consistency, stability over time, and equivalence of alternate forms. Internal consistency examines consistency of performance across parts of a test and can be estimated using split-half reliability, which treats halves of a test as parallel forms.
The document discusses developing materials for language courses. It defines developing materials as the process teachers use to create lessons and units to meet course goals and objectives. Teachers must consider factors like course goals, their teaching philosophy, and how students learn when developing materials. Well-designed activities should engage students, build skills and confidence, integrate different skills, and use authentic texts when possible. The document provides examples of materials developed by teachers for courses on topics like family, writing, and relationships. It emphasizes that developing effective materials depends on understanding teachers' beliefs and students' needs in the specific context.
The document discusses different types of language tests, including objective tests, subjective tests, direct tests, indirect tests, discrete-point tests, and integrative tests. It also discusses the differences between norm-referenced tests and criterion-referenced tests. Norm-referenced tests measure students against each other, while criterion-referenced tests measure students against an absolute standard. The key differences are that norm-referenced tests have a broader content focus, aim to discriminate between high and low achievers, and report scores in percentiles compared to peers. Criterion-referenced tests have a specific content domain, measure whether students achieve specific skills, and report scores as a percentage of skills mastered.
This document discusses teaching listening skills in English as a second language. It begins by stating that the objective of classroom listening is to prepare students for real-life listening situations. It then lists common features of real-life listening situations, such as informal language, noise, redundancy, and ongoing listener response.
The document notes that classroom listening does not replicate real-life listening. It recommends basing listening activities on simulated real-life situations to motivate students. Guidelines are provided for listening texts, such as using informal speech, and for listening tasks, such as providing expectations and an ongoing purpose. Finally, examples are given of different types of listening activities and how they can be adapted.
Testing writing (for Language Teachers)Wenlie Jean
The document discusses the key considerations for properly testing writing ability. It identifies four main problems in testing: 1) using representative tasks, 2) eliciting valid writing samples, 3) ensuring scores are valid and reliable, and 4) providing feedback. For each, it outlines various factors that test designers should take into account such as specifying all content domains, including a representative task sample, restricting candidates, using appropriate scoring scales, and calibrating scorers. The goal is to develop writing tests that accurately measure students' abilities.
This document summarizes key aspects of curriculum design approaches from chapters 9 and 10 of the book "Language Curriculum Design" by I.S.P Nation and John Macalister. It discusses three common approaches to the curriculum design process: the waterfall model, focused opportunistic approach, and layers of necessity model. It also covers negotiated syllabuses, where teachers work with learners to make joint decisions about curriculum design elements. Requirements for implementing a negotiated syllabus include establishing negotiation procedures, planning course content and activities, setting learning goals, and evaluating outcomes.
The document discusses assessing writing skills. It describes different types of writing like academic, job-related, and personal writing. It outlines micro skills like imitative and intensive writing, and macro skills like responsive and extensive writing. For micro skills, it provides examples of assessment tasks for imitative writing like spelling tests and dictation. For intensive writing, it discusses tasks like rewriting sentences and transforming grammar. For macro skills, it discusses designing assessment tasks for responsive and extensive writing like guided questions, paragraph construction, and scoring methods.
Language testing is the practice of evaluating an individual's proficiency in using a particular language. There are two main types of assessment: formative assessment which checks student progress, and summative assessment which measures achievement at the end of a term. There are five common types of language tests: proficiency tests which measure overall ability, achievement tests related to course content, diagnostic tests which identify strengths and weaknesses, placement tests for assigning students to class levels, and direct/indirect tests. The effect of testing on teaching is known as backwash, which can be harmful if not aligned with course objectives, or beneficial if tests influence instructional changes.
This document provides an overview of linguistic approaches to analyzing ideology between 1979 and 2010. It discusses three main approaches: Critical Linguistics (CL), Critical Discourse Analysis (CDA), and Critical Stylistics (CS). CL, introduced in 1979, used linguistic analysis and Systemic Functional Grammar to analyze ideology in texts. CDA, developed in 1989, expanded CL to include social and institutional contexts. Between 1989-2010, various CDA theories emerged. CS, introduced in 2010, returned linguistic analysis to the forefront by drawing on CL and CDA. The document reviews developments across these three decades in applying linguistic theories to analyze textual ideologies.
This document discusses assessment in language learning. It defines assessment as collecting information about students' language development through various methods such as tests, portfolios, and observations. This information is then analyzed and used to make pedagogical decisions. For assessment to be effective, it should be valid, reliable, and feasible. Validity means the assessment accurately measures proficiency. Reliability means a student would achieve similar results on multiple attempts. Feasibility means the assessment is practical to implement. The document also discusses formative assessment, self-assessment, and the importance of feedback in the learning process.
This document discusses different types of language tests and testing, including proficiency tests, achievement tests, diagnostic tests, placement tests, direct and indirect testing, discrete point and integrative testing, norm-referenced and criterion-referenced testing, objective and subjective testing, and computer adaptive testing. It provides details on the purpose and characteristics of each type of test.
This document distinguishes between language systems and language skills. It discusses the key components of language systems, including phonology, lexis, grammar, morphology, syntax, and functional and situational aspects. It also outlines the four main language skills - listening, speaking, reading, and writing - and how they relate to each other as receptive or productive skills involving spoken or written communication. The document emphasizes that an integrated approach to teaching language systems and skills is important for effective language learning and communication.
This document discusses key concepts in language assessment including validity, reliability, and feasibility. It defines validity as the accuracy of a test in measuring the intended proficiency. There are different types of validity including content, criterion-related, and construct validity. Reliability refers to a test producing consistent results, which can be measured using methods like test-retest. Feasibility means a test is practical to administer. The document also discusses types of language tests, how to improve validity and reliability, and item analysis. Chapters from a book on language testing techniques are assigned for discussion.
The Common European Framework of Reference (CEFR) is a standard developed by the Council of Europe to describe language ability. It introduces six common reference levels (A1 to C2) to standardize language education across Europe. The CEFR provides clear definitions of what language learners can do at each level to facilitate cooperation in language education.
CEFR-based tools and resources: latest developments (Mila Angelova)eaquals
This document provides an overview of an EAQUALS session on CEFR-based curriculum and assessment. It discusses developing resources like the Core Inventory for French and reading/listening scenarios. The EAQUALS Certificate of Achievement scheme guarantees quality CEFR-implemented assessment and curriculum design through screening processes. Benefits include differentiating members and demonstrating academic competence. Main prerequisites for certification include a CEFR-based curriculum, standardization training, and moderation techniques. The session aims to help members implement CEFR-based approaches and identify areas of interest.
This document summarizes key points about testing various language skills from Hughes' book on language testing. It discusses techniques for testing writing, oral abilities, reading, listening, grammar, vocabulary, and overall ability. Tips are provided for each skill area as well as for testing young learners. Common issues with indirect assessment are addressed. The importance of test design, task selection, task difficulty, and reliable scoring procedures are emphasized throughout.
This document discusses strategies for assessing and grading students according to the Common European Framework of Reference (CEFR). It recommends using CEFR criteria for low-stakes, in-class assessment but standardized exams for high-stakes certification. When assessing CEFR levels, schools should relate their curriculum to CEFR descriptors and develop assessment tasks aligned to the descriptors. Moderation is important to counteract subjectivity and ensure consistent standards are applied. Criterion-referenced assessment according to CEFR criteria provides explicit information about students' abilities independent of peer performance.
The Common European Framework of Reference for Languages (CEFR) provides a common basis for describing language learning, teaching and assessment. It was developed by the Council of Europe between 1989-1996. The main purposes of the CEFR are to provide a method for teaching, learning and assessing languages, and to set common standards so that language qualifications can be recognized across Europe. It divides learners into 6 levels of proficiency and defines the language skills students should develop at each level, including understanding, speaking, and writing.
The Common European Framework of Reference for Languages (CEFR) provides a common basis for describing language ability across Europe. It describes what language learners need to know and be able to do to use a language for communication. The CEFR defines six reference levels of language proficiency from A1 for basic users to C2 for mastery. It also outlines the grammatical structures and competencies required at each level. The CEFR takes a communicative approach, focusing on learners' needs and basing teaching on developing communicative competence through everyday interactions and cultural understanding.
The document discusses different types of assessments including formative assessment, which is used to identify if students have achieved the lesson objective and determine gaps. Examples of formative assessments include questioning students and collecting assignments. Summative assessment provides grades based on performance over a period of time, such as final exams. Performance assessment evaluates what students can do in real-world scenarios through demonstrations and projects.
This document discusses different types of assessment and evaluation tools used in education. It describes diagnostic, formative, and summative assessments and their purposes. Diagnostic assessments identify student strengths and weaknesses at the start of instruction. Formative assessments evaluate student learning throughout instruction to help students improve. Summative assessments make judgments about student achievement at the end of a learning period. The document also outlines specific tools like observations, checklists, interviews, and projects that can be used for assessment and evaluation.
The document discusses assessment practices and formative assessment. It provides an overview of assessment types including formative, summative, and diagnostic assessments. Formative assessment identifies student needs, guides ongoing instruction, and provides feedback to improve learning, while summative assessment evaluates learning at the end of a unit. The document emphasizes that formative assessment, when used to adapt teaching to meet student needs, has a strong positive effect on learning.
This document discusses different types of assessment used to evaluate learner performance. Formal assessment includes tests and exams with numerical grades, while informal assessment observes learner skills through comments without grades. Self-assessment and peer assessment also allow learners to evaluate themselves and each other. Examples of assessment tasks that can be used formally or informally include gap fills, multiple choice questions, interviews, compositions and dictation. Tasks vary in what they measure, from communication skills to language accuracy, and in how easy they are to score objectively versus subjectively. Informal methods include observation, note-taking and self/peer evaluation sheets.
This document outlines various topics related to language testing, including types of tests, approaches to testing, validity and reliability, and achieving beneficial backwash effects. It discusses proficiency tests, achievement tests, and diagnostic tests. It also covers direct and indirect testing, norm-referenced and criterion-referenced testing, and objective and subjective testing. Validity is defined as accurately measuring the intended abilities, while reliability is consistency of results. Achieving beneficial backwash means testing abilities you want to foster and ensuring students and teachers understand the test.
This document provides an outline for a course on testing for language teachers. It covers various topics related to language testing including the purposes of different types of tests, approaches to testing, ensuring validity and reliability, and achieving beneficial backwash effects. The key points covered are the types of tests (proficiency, achievement, diagnostic, placement), approaches to testing (direct vs indirect, discrete point vs integrative), factors of validity and reliability, and how to design tests that motivate effective teaching practices.
This document discusses assessment in learning and various types of tests used to evaluate student learning. It describes different ways to classify tests, such as oral versus written tests, and objective versus subjective tests. The document outlines the key characteristics of formative, summative, diagnostic, and standardized tests. It also discusses important criteria for developing good tests, such as validity, reliability, and objectivity. Different scoring methods and statistical measures used to analyze test results are presented, such as frequency distributions and measures of central tendency like the mean.
This document provides an overview of language testing. It discusses the types of language tests, including proficiency tests, achievement tests, and placement tests. It also covers approaches to testing such as direct vs indirect testing, discrete point vs integrative testing, and norm-referenced vs criterion-referenced testing. The document emphasizes the importance of test validity and reliability. It provides tips for developing valid and reliable tests, such as ensuring clear instructions, unambiguous questions, and objective scoring. The goal of language testing should be to accurately measure language abilities and encourage beneficial learning.
This document discusses the key characteristics of effective assessment: validity, reliability, practicality, and accuracy. It defines each characteristic and provides examples. Validity means a test measures what it intends to measure. Reliability means a test produces consistent results. Practicality means a test is usable in terms of time and cost. Accuracy means a test is free from errors. The document also discusses factors that affect the acceptability of a test like length, technique, administration conditions, and presentation quality. Overall, the document provides an overview of the essential features of assessment and testing.
This document summarizes key aspects of language testing based on a course presentation. It discusses the need for valid and reliable tests, and outlines different types of tests including proficiency, achievement, and placement tests. It also covers approaches to testing such as direct vs indirect, discrete point vs integrative, and norm-referenced vs criterion-referenced testing. The document emphasizes the importance of test validity and reliability, and provides tips for improving tests.
This document discusses key concepts and principles of assessment for English language learners. It begins by explaining why assessment should take place, noting that it is used to measure learning and improve instruction. It then covers key concepts involved in assessment like accountability, achievement, and different assessment types and strategies. Several principles of assessment are outlined, including being ethical, fair, valid, reliable and practical. The document concludes by providing checklists to evaluate if classroom tests are applying these principles of practicality, reliability, validity, authenticity, and having a beneficial washback effect on learning.
The document discusses principles of testing including practicality, reliability, validity, and different types of tests. It addresses how to make tests more reliable and valid. Reliability refers to consistency and dependability, and can be improved through clear instructions, uniform conditions, and objective scoring. Validity means a test accurately measures what it intends to. Communicative competence and practical issues in testing are also covered.
This document provides information about English proficiency tests and the process of constructing and standardizing such tests. It discusses two common proficiency tests, IELTS and TOEFL, outlining their testing components and procedures. Key aspects of test construction addressed include defining objectives, developing and reviewing test items, pretesting items, and ensuring questions are unbiased. The document also outlines the steps in standardizing tests, such as assembling the test, statistical analysis of items, and reliability reviews. Item analysis is described as a method to evaluate how well individual test questions are performing.
Testing and Test construction (Evaluation in EFL)Samcruz5
The document discusses different types of tests, their purposes, and characteristics. It describes screening, placement, proficiency, aptitude, diagnostic, achievement, and progress tests. Direct and indirect tests as well as discrete point and integrative tests are also defined. Guidelines for developing effective tests are provided, including moderating tasks, determining an appropriate level of difficulty, ensuring discrimination among learners, using a representative sample, avoiding overlap, providing clear instructions, considering timing, and avoiding bias in the layout. Moderation of marking schemes, standardization of examiners, and follow-up reviews are also recommended to help ensure tests achieve their intended purposes.
Testing and Test Construction (Evaluation ILE)Samcruz5
Testing serves several purposes including informing learners and teachers of strengths and weaknesses, motivating learners, and determining if learning objectives have been achieved. Tests can be classified based on their purpose such as screening, placement, or achievement. They can also be classified based on their characteristics such as being direct or indirect, criterion-referenced or norm-referenced. Proper test design considers guidelines around moderating tasks, ensuring an appropriate level of difficulty, avoiding bias, and standardizing examiners.
This document discusses standardized tests and test construction. It defines standardized tests as tests where all students answer the same questions in the same way, allowing performance to be compared. The main types of standardized tests are norm-referenced tests, which compare performance to others, and criterion-referenced tests, which compare performance to objectives. Good test construction involves planning test objectives, writing clear and valid questions, and revising the test based on analysis to ensure it reliably measures the desired content.
Validity refers to a test accurately measuring what it intends to. Content validity means a test samples relevant skills, while criterion-related validity compares test scores to external criteria. Reliability means a test gives consistent results. Key factors for reliability include multiple test items, clear instructions, uniform administration conditions, and scorer reliability through objective scoring and scorer training. While reliability ensures consistent results, a test may be reliable without being valid if it does not accurately measure the target construct. Both validity and reliability are important for effective test design and interpretation.
This document discusses various concepts related to educational measurement and testing. It defines key terms like measurement, evaluation, tests, validity, reliability, and different types of tests. It provides details on constructing objective tests, including writing test items, establishing test validity and reliability, and interpreting test scores. It also discusses advantages and disadvantages of objective and essay tests. Additionally, it covers topics like measures of central tendency, frequency distributions, and calculating the mean, median and mode.
The document discusses developing assessment instruments for measuring learner progress and instructional quality. It covers criterion-referenced assessments that measure performance against specific standards or levels. The objectives are to describe criterion-referenced tests and different types of pre- and post-instruction assessments. It also discusses developing quality criterion-referenced test items and assessments of products, performances, and attitudes.
The document discusses developing assessment instruments for measuring learner progress and instructional quality. It describes criterion-referenced assessments that measure performance against specific standards or levels of mastery. The objectives are to describe criterion-referenced tests and how various assessment types (entry tests, pretests, practice tests, posttests) are used. It also discusses developing quality criterion-referenced test items in four categories: goal-centered, learner-centered, context-centered, and assessment-centered.
The document discusses developing criterion-referenced assessments. It explains that criterion-referenced assessments directly measure skills described in behavioral objectives and focus on gauging learner performance and instructional quality. The document provides guidance on writing test items, developing different types of assessments, setting mastery criteria, and ensuring assessments are congruent with objectives and instructional analyses. It emphasizes the importance of criterion-referenced assessments for evaluating both learners and instruction.
This document provides guidelines for constructing different types of written tests to assess student learning. It begins by outlining the desired learning outcomes, which are to identify appropriate test formats for different outcomes and apply guidelines for constructing test items. It then describes various test formats, including selected response (e.g. multiple choice) and constructed response (e.g. essays, short answer). The document provides detailed guidelines for writing high-quality test items for multiple choice, matching, and true/false question formats. Teachers are advised to choose formats based on learning outcomes and cognitive level, and to write clear stems and options to develop valid and reliable assessments of student knowledge.
This document provides an overview of standardized and non-standardized tests. It defines standardized tests as those with uniform administration and scoring to allow for comparison. Examples include achievement, IQ, and aptitude tests. Non-standardized tests are constructed by teachers and vary in their administration. The document discusses the construction, uses, and limitations of different types of test items for standardized and non-standardized tests such as essay, short answer, multiple choice, and matching questions. Rating scales and checklists are also covered as methods for assessment.
This document discusses bilingual education programs at higher education institutions that use English as the primary language of instruction, known as EMI programs. It notes the increasing trend of EMI programs in Europe and reasons for their growth, including internationalization, improving English skills, and prestige. Potential threats of EMI discussed include lack of English proficiency among students and teachers leading to ineffective teaching and learning, and EMI limiting classroom discourse. Solutions proposed include screening language levels, additional training, and bilingual degrees. Research on EMI programs found small improvements in students' English skills. Examples of EMI programs in Spanish universities are also provided.
This document outlines 3 modules for language teaching. Module 1 provides background information on language teaching. Module 2 covers lesson planning and use of resources. Module 3 focuses on managing the teaching and learning process in the classroom.
The document provides an overview of the geography and regions of the United States. It describes the country's large scale and varied landscape and climate. The US is divided into several distinct regions, including New England, the Mid-Atlantic region, the South, the Midwest, the Southwest, the Rocky Mountain region, and Pacific states. Each region has its own history and culture, and the document profiles some of the major cities and landmarks within each one.
Early Britain saw the construction of Stonehenge and Celtic population during the Iron Age. The Romans invaded in 43 AD and built Hadrian's Wall to separate Roman and barbarian tribes. Anglo-Saxons then invaded in the 5th-6th centuries, establishing kingdoms and converting to Christianity under the influence of Celtic and Roman faiths. The Normans conquered England in 1066 under William the Conqueror, establishing French dominance. The Magna Carta limited royal power in 1215 and the Hundred Years' War with France began in 1337. The Tudors rose to power in 1485 and Henry VIII established the Church of England in the 1530s. Civil war erupted in 1642 over disputes
This document discusses how to teach English literature in the classroom. It defines literature and English literature, explaining that literature includes stories, poems, and plays considered to have artistic value. It recommends using a learner-centered approach that encourages personal growth and interaction with texts. Teachers should select texts that are appropriate for their students' ages, English levels, and learning objectives. A variety of activities are described to help students engage with texts both inside and outside the classroom, including using comics due to their motivational value and ease of understanding.
The European Association for Language Testing and Assessment (EALTA) aims to promote understanding of language testing principles and improve testing practices across Europe. EALTA's guidelines provide best practices for those involved in teacher training, classroom assessment, and test development. The guidelines stress respect, responsibility, fairness, reliability, and validity. They also recommend clarifying purposes and ensuring appropriateness, accuracy, feedback, and stakeholder involvement in the assessment process. EALTA encourages engagement with decision makers to enhance quality of assessment systems.
Culture can be defined in several ways:
1. Traditionally, culture referred to civilization as opposed to nature, including high art and intellectual achievements.
2. Anthropologists view culture as a worldview comprising the knowledge, beliefs, arts, and customs acquired by people as members of a society.
3. Cultural anthropologists see culture as systems of symbols people use to communicate and understand each other.
Culture encompasses the distinctive spiritual, material, intellectual and emotional features of a society or group, including art, literature, lifestyles, beliefs and values.
The document outlines an action-oriented language model used in Cantabria, Spain for teaching language. The model uses task-based learning principles including using authentic materials and real-life tasks to reproduce natural language acquisition. Some key methodological principles are integrating skills and contents through communicative tasks, using texts close to students' experiences, promoting student autonomy, and treating errors positively as part of the learning process. An example task is provided focusing on family topics, with associated learning goals and activities involving listening, reading, writing, and oral interaction.
The document describes the European Language Portfolio (ELP), which aims to help language learners improve their learning process through self-assessment and reflection. It discusses the ELP's objectives, sections (passport, biography, dossier), and types for different age groups in Spain. It then details the implementation of ELPs at two Spanish schools, including developing activities, trials with students, and addressing problems like the difficulty of self-assessment and the ELP's bulkiness. Teachers found that ELPs increased student autonomy and awareness of language learning as a process. The document concludes by discussing ways ELP principles have been incorporated into language courses, and the impact of ELPs on teacher training.
This document provides instructions for a training program to familiarize users with the Common European Framework of Reference for Languages (CEF). The training involves 6 steps: 1) selecting a language skill, 2) choosing a communicative task, 3) reading the task description, 4) completing the task, 5) rating the task difficulty, and 6) checking the user's rating against the trainer's rating. The goal is to help users scale language skills and assess task and performance levels as defined by the CEF.
The Common European Framework provides a common basis for language education across Europe by establishing common reference levels for languages. It aims to promote plurilingualism, lifelong learning, and greater mobility and cooperation through common standards. The Framework describes language ability through communicative competences and sets out descriptive levels from A1 for basic users up to C2 for mastery. It takes a comprehensive approach to language skills including reception, production, interaction and mediation.
This document outlines the theoretical frameworks and main schools of thought regarding the relationship between film and literature. It discusses the precursors to film theory from the avant-garde, Russian formalism, and early 20th century writers. It also summarizes the influential semiological and narratological studies from thinkers like Christian Metz, Umberto Eco and Roland Barthes. Finally, it examines some of the key linguistic differences between film and literature, such as their approaches to space, time, and point of view, as well as some extra-linguistic aspects like their modes of creation and reception.
This document discusses different perspectives for analyzing popular fiction, including Dashiell Hammett's hard-boiled detective novels. It covers views of the author, reader, social context, genre codes, and medium. Regarding Hammett's works specifically, it notes they were successful on both a casual and careful reading level, offering absorbing stories and rich language. The document also examines criteria for classifying crime fiction genres, such as detective, noir, and thriller.
Dashiell Hammett was an American author best known for hardboiled crime novels and short stories published in the 1920s and 1930s, which had a significant influence on subsequent crime writers. Some of his most famous works include The Maltese Falcon, The Thin Man, and Red Harvest. He pioneered the "hardboiled" school of detective fiction, featuring private detectives who are often more concerned with honor than obedience to the law.
This document discusses definitions of Europe's boundaries. It notes that Europe's eastern boundary is commonly defined as the Ural Mountains, while its southeastern boundary with Asia is less clearly defined, possibly being the Ural River, Emba River, or Caucasus Mountains. The Mediterranean Sea separates Europe from Africa, and the Atlantic Ocean separates Europe from North America. However, there are differing opinions on where boundaries fall. Some include parts of Western Asia like Turkey and the Caucasus region within Europe, while others define boundaries along geographic features. The document also lists some transcontinental countries located in both Europe and Asia.
The document discusses language proficiency levels based on the Common European Framework and strategies for improving English skills at a university. It outlines 6 proficiency levels from A1 to C2 and provides examples of certifications and exams. It then proposes 3 solutions for students to demonstrate an B2 level: taking additional English classes, having subjects taught in English, or obtaining other qualifications involving English study.
El documento presenta un resumen del Plan Curricular del Instituto Cervantes de 2006. Establece tres objetivos generales centrados en el alumno como agente social, hablante intercultural y aprendiente autónomo. Además, describe los cinco componentes de la lengua que comprende el plan: gramatical, pragmático-discursivo, nocional, cultural y de aprendizaje. Finalmente, ofrece ejemplos de los inventarios de los componentes gramatical, pragmático-discursivo y cultural.
El documento describe dos aplicaciones principales del Marco Común Europeo de Referencia para las Lenguas: DIALANG, un sistema de evaluación de conocimientos de idiomas en línea disponible para 14 idiomas y tres destrezas, y Europass, un conjunto de documentos que ayudan a comunicar cualificaciones y titulaciones para facilitar la movilidad en la Unión Europea. También menciona algunos otros recursos en línea relacionados con el aprendizaje y evaluación de idiomas.
El documento describe los conceptos clave de la evaluación según el MCER: validez, fiabilidad y viabilidad. Explica que para que la evaluación sea válida, fiable y viable, es necesario precisar lo que se evalúa (actividades comunicativas), cómo se interpreta la actuación (criterios de evaluación), y seguir códigos de buenas prácticas. Además, enumera 13 tipos diferentes de evaluación.
Este documento presenta la metodología para nuevos currículos de idiomas basados en el Marco Común Europeo de Referencia para las Lenguas. Describe 11 principios metodológicos como el uso de tareas comunicativas, la integración de destrezas y contenidos, y textos cercanos a la experiencia del estudiante. También incluye ejemplos de objetivos, actividades y recursos para el nivel B1.
Chapter wise All Notes of First year Basic Civil Engineering.pptxDenish Jangid
Chapter wise All Notes of First year Basic Civil Engineering
Syllabus
Chapter-1
Introduction to objective, scope and outcome the subject
Chapter 2
Introduction: Scope and Specialization of Civil Engineering, Role of civil Engineer in Society, Impact of infrastructural development on economy of country.
Chapter 3
Surveying: Object Principles & Types of Surveying; Site Plans, Plans & Maps; Scales & Unit of different Measurements.
Linear Measurements: Instruments used. Linear Measurement by Tape, Ranging out Survey Lines and overcoming Obstructions; Measurements on sloping ground; Tape corrections, conventional symbols. Angular Measurements: Instruments used; Introduction to Compass Surveying, Bearings and Longitude & Latitude of a Line, Introduction to total station.
Levelling: Instrument used Object of levelling, Methods of levelling in brief, and Contour maps.
Chapter 4
Buildings: Selection of site for Buildings, Layout of Building Plan, Types of buildings, Plinth area, carpet area, floor space index, Introduction to building byelaws, concept of sun light & ventilation. Components of Buildings & their functions, Basic concept of R.C.C., Introduction to types of foundation
Chapter 5
Transportation: Introduction to Transportation Engineering; Traffic and Road Safety: Types and Characteristics of Various Modes of Transportation; Various Road Traffic Signs, Causes of Accidents and Road Safety Measures.
Chapter 6
Environmental Engineering: Environmental Pollution, Environmental Acts and Regulations, Functional Concepts of Ecology, Basics of Species, Biodiversity, Ecosystem, Hydrological Cycle; Chemical Cycles: Carbon, Nitrogen & Phosphorus; Energy Flow in Ecosystems.
Water Pollution: Water Quality standards, Introduction to Treatment & Disposal of Waste Water. Reuse and Saving of Water, Rain Water Harvesting. Solid Waste Management: Classification of Solid Waste, Collection, Transportation and Disposal of Solid. Recycling of Solid Waste: Energy Recovery, Sanitary Landfill, On-Site Sanitation. Air & Noise Pollution: Primary and Secondary air pollutants, Harmful effects of Air Pollution, Control of Air Pollution. . Noise Pollution Harmful Effects of noise pollution, control of noise pollution, Global warming & Climate Change, Ozone depletion, Greenhouse effect
Text Books:
1. Palancharmy, Basic Civil Engineering, McGraw Hill publishers.
2. Satheesh Gopi, Basic Civil Engineering, Pearson Publishers.
3. Ketki Rangwala Dalal, Essentials of Civil Engineering, Charotar Publishing House.
4. BCP, Surveying volume 1
This presentation was provided by Rebecca Benner, Ph.D., of the American Society of Anesthesiologists, for the second session of NISO's 2024 Training Series "DEIA in the Scholarly Landscape." Session Two: 'Expanding Pathways to Publishing Careers,' was held June 13, 2024.
🔥🔥🔥🔥🔥🔥🔥🔥🔥
إضغ بين إيديكم من أقوى الملازم التي صممتها
ملزمة تشريح الجهاز الهيكلي (نظري 3)
💀💀💀💀💀💀💀💀💀💀
تتميز هذهِ الملزمة بعِدة مُميزات :
1- مُترجمة ترجمة تُناسب جميع المستويات
2- تحتوي على 78 رسم توضيحي لكل كلمة موجودة بالملزمة (لكل كلمة !!!!)
#فهم_ماكو_درخ
3- دقة الكتابة والصور عالية جداً جداً جداً
4- هُنالك بعض المعلومات تم توضيحها بشكل تفصيلي جداً (تُعتبر لدى الطالب أو الطالبة بإنها معلومات مُبهمة ومع ذلك تم توضيح هذهِ المعلومات المُبهمة بشكل تفصيلي جداً
5- الملزمة تشرح نفسها ب نفسها بس تكلك تعال اقراني
6- تحتوي الملزمة في اول سلايد على خارطة تتضمن جميع تفرُعات معلومات الجهاز الهيكلي المذكورة في هذهِ الملزمة
واخيراً هذهِ الملزمة حلالٌ عليكم وإتمنى منكم إن تدعولي بالخير والصحة والعافية فقط
كل التوفيق زملائي وزميلاتي ، زميلكم محمد الذهبي 💊💊
🔥🔥🔥🔥🔥🔥🔥🔥🔥
Leveraging Generative AI to Drive Nonprofit InnovationTechSoup
In this webinar, participants learned how to utilize Generative AI to streamline operations and elevate member engagement. Amazon Web Service experts provided a customer specific use cases and dived into low/no-code tools that are quick and easy to deploy through Amazon Web Service (AWS.)
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.pptHenry Hollis
The History of NZ 1870-1900.
Making of a Nation.
From the NZ Wars to Liberals,
Richard Seddon, George Grey,
Social Laboratory, New Zealand,
Confiscations, Kotahitanga, Kingitanga, Parliament, Suffrage, Repudiation, Economic Change, Agriculture, Gold Mining, Timber, Flax, Sheep, Dairying,
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.ppt
7 assessment and the cefr
1.
2. Assessment: Assessment of the proficiency of
the language user
3 key concepts:
• Validity: the information gained is an accurate
representation of the proficiency of the candidates
• Reliability: A student being tested twice will get the
same result (technical concept: the rank order of the
candidates is replicated in two separate—real or
simulated—administrations of the same assessment )
• Feasibility: The procedure needs to be practical,
adapted to the available elements and features
3. If we want assessment to be valid, reliable,
and feasible, we need to specify:
• What is assessed: according to the CEFR,
communicative activities (contexts, texts, and tasks).
See examples.
• How performance is interpreted: assessment criteria.
See examples
• How to make comparisons between different tests
and ways of assessment (for example, between public
examinations and teacher assesment). Two main
procedures:
Social moderation: discussion between experts
Benchmarking: comparison of samples in relation to
standardized definitions and examples, which become
reference points (benchmarks)
• Guidelines for good practice: EALTA
5. Types of tests:
• Proficiency tests
• Achievement tests. 2 approaches:
To base achievement tests on the textbook/syllabus
To base them on course objectives. More beneficial
washback.
• Diagnostic tests
• Placement tests
6. Validity Types:
• Construct validity (very general, the information
gained is accurate representation of the
proficiency of the candidate. It checks the validity
of the construct, the thing we want to measure)
• Content validity. This checks it the test’s content is
a representative simple of the skills or structures
that it wants to measure. In order to check this we
need a complete specification of all the skills or
structures we want to cover. If it covers 5% only, it
has less content validity than if it covers 25 %.
7. Validity Types:
• Criterion-related validity: Results on the test agree with
other dependable results (criterion test)
Concurrent validity. We compare the test results with the
criterion test.
Predictive validity. A placement test is validated by the
teachers who teach the selected students.
• Validity in scoring. Not only the items need to be valid,
but also the way in which responses are scored
(taking into account grammar mistakes in a reading
comprehension exam is not valid)
• Face validity: the test has to look as if it measures
what it is supposed to measure. A written test to check
pronunciation has little face validity.
8. How to make tests more valid (Hughes)
Write specifications for the test.
Include a representative sample ot the
content of the specifications in the text
Whenever feasible, use direct testing
Make sure that the scoring relates directly
to what is being tested
Try to make the test reliable
9. Reliability: A student being tested twice will get the same
result (technical concept: the rank order of the candidates
is replicated in two separate—real or simulated—
administrations of the same assessment )
- We compare two tests. Methods:
- Test-Retest: the student takes the same test again
- Alternate Forms: the students take two alternate forms
of the same test
- Split.Half: you split the test into two equivalent halves
and compare them as if they were two different tests.
10. - Reliability coefficient / Standard Error of Measurement
A High Stakes Test needs a high reliability coefficient
(highest is 1), and therefore a very low standard error of
measurement (a number obtained by statistical analysis). A
Lower Stakes exam does not need those coefficients.
- True Score: the real score that a student would get in a
perfectly reliable test. In a very reliable test, the true
score is clearly defined (the student will always get a
similar result, for example 65-67). In a less reliable test,
the range is wider (55-75).
- Scorer reliability (coefficient). You compare the scores
given by different scorers (examiners). The more
agreement, the more reliable their reliability coefficient.
11. Item analysis:
Facility value
Discrimination indices: drop some, improve
others
Analyse distractors
Item banking
12. 1.Take enough samples of behaviour.
2.Exclude items which do not descriminate well
3.Do not allow candidates too much freedom.
4.Write unambiguous items
5.Provide clear and explicit instructions
6.Ensure that tests are well laid out and perfectly
legible
7.Make candidates familiar with format and testing
techniques
8.Provide uniform and non-distracting conditions of
administration
13. 9. Use items which permit scoring which is as
objective as possible
10. Make comparisons between candidates as direct
as possible
11. Provide a detailed scoring key
12. Train scorers
13. Agree acceptable responses and appropriate
scores at the beginning of the scoring process.
14. Identifty candidates by number not by name
15. Employ multiple, independent scorers..
14. To be valid a test must be reliable (provide
accurate measurement)
A reliable test may not be valid at all
(technically perfect, but globally wrong: it
does not test what it is supposed to test)
15. Washback/Backwash
Test the abilities/skills you want to encourage.
Sample widely and unpredictably
Use direct testing
Make testing criterion-referenced (CEFR)
Base achievement tests on objectives
Ensure that the test is known and understood by
students and teachers
Counting the cost
16. 1. Make a full and clear statement of the testing
‘problem’.
2. Write complete specifications for the test.
3. Write and moderate items.
4. Trial the items informally on native speakers
and reject or modify problematic ones as
necessary.
5. Trial the test on a group of non-native
speakers similar to those for whom the test is
intended.
6. Analyse the results of the trial and make any
necessary changes.
7. Calibrate scales.
8. Validate.
9. Write handbooks for test takers, test users
and staff.
10. Train any necessary staff (interviewers,
raters, etc.).
18. Chapters from Hughes’ Testing for Language Teachers
9. Testing Writing
10. Testing Oral Abilities
11. Testing Reading
12. Testing Listening
13. Testing Grammar and Vocabulary
14. Testing Overall Ability
15. Tests for Young Learners
19. 1. Set representative tasks
1. Specify all possible content
2. Include a representative sample of the specified content
2. Elicit valid samples of writing ability
1. Set as many separate tasks as feasible
2. Test only writing ability and nothing else
3. Restrict candidates
3. Ensure valid and reliable scoring:
1. Set as many tasks as possible
2. Restrict candidates
3. Give no choice of tasks
4. Ensure long enough samples
5. Create appropriate scales for scoring: HOLISTIC/ANALYTIC
6. Calibrate the scale to be used
7. Select and train scorers
8. Follow acceptable scoring procedures
20. • “The most highly prized language skill”, Lado’s
Language Testing (1961).
• Challenges: ephemeral, intangible.
• Contrast US/UK: Certificate of Proficiency in English
(1913) already included it, TOEFL only in 2005 iBT
• Key notion: not accent, but intelligibility
• Very different approaches.
Indirect
Direct (Cambridge, EOIs) or Semi-direct (TOEFL ibt, OTE,
Aptis). Conflict with the American tradition.
The future?: Fully automated L2 speaking tests: Versant
test, Speechrater.
• Not only speaking, also interaction
21. 1. Set representative tasks
1. Specify all possible content
2. Include a representative sample of the specified content
2. Elicit valid samples of oral ability.
1. Techniques:
1. Interview :Questions, pictures, role play, interpreting (L1 to L2),
prepared monologue, reading aloud
2. Interaction: discussion, roleplay
3. Responses to audio- or video-recordings (semi-direct)
2. Plan and structure the test carefully
1. Make the oral test as long as it is feasible
2. Plan the test carefully
3. As many tasks (“fresh starts”) as possible
4. Use a second tester
5. Set only tasks that candidates could do easily in L1
22. Plan and structure the test carefully
1. Set only tasks that candidates could do easily in L1
2. Quiet room with good acoustics
3. Put candidates at ease (at first, easy questions, not assessed,
problem with note-taking?)
4. Collect enough relevant information
5. Do not talk too much
6. (select interviewers carefully and train them)
1. Ensure valid and reliable scoring:
1. Create appropriate scales for scoring: HOLISTIC/ANALYTIC. Calibrate
the scale to be used
2. Select and train scorers (different from interviewers if possible)
3. Follow acceptable scoring procedures
23. PROBLEMS:
Indirect assessment:
We read in very different ways: scanning, skimming,
inferring, intensive, extensive reading…
SOME TIPS
As many texts and operations as possible (Dialang).
Avoid texts which deal with general knowledge
Avoid disturbing topics, or texts students might have
read
Use authentic texts
Techniques: better short answer and gap filling than
multiple choice
Task difficulty can be lower than text difficulty
Items should follow the order of the text
Make items independent of each other
Do not take into account errors of grammar or spelling
24. PROBLEMS
As in listening: Indirect assessment and different ways
of listening
As in speaking: Transient nature of speech
http://www.usingenglish.com/articles/why-your-students-have-problems-with-listening-comprehension.html
TIPS:
Same as in reading
If recording is used, make it as natural as possible
Items should be far apart in the text
Give students time to become familiar with the tasks
Techniques: apart from multiple choice, shot answers
and gap filling, information transfer, note taking, partial
dictation, transcription
Moderation is essential
How many times?
25. GRAMMAR
Why? Easy to test, Content validity
Why not? Harmful washback effect
It depends on the type of test.
Specifications: from the Council of Europe books
Techniques: Gap filling, rephrasings, completion
Don’t penalize for mistakes that were not tested (-s if
the item is testing relatives, for example)
VOCABULARY
Why (not)?
Specifications: use frequency considerations
Techniques:
Recognition: Recognise synonims, recognise definitions,
recognise appropriate word for context
Production: pictures, definitions, gap filling,
26. Useful in particular tests where washback is
not important
Cloze test (from closure). Based on the
idea of “reduced redundancy”. Subtypes:
Selected deletion cloze
Conversational cloze
C-Tests
Dictation
Main problem : horrible washback effect.
27. TIPS
- Testing-assessment-teaching
- Feedback
- Self assessment
- Washback
- Short tasks
- Use stories and games
- Use pictures and color
- Don’t forget that children are still developing L1 and cognitive
abilities
- Include interaction
- Use colour and drawing
- Use cartoon stories
- Long warm-ups in speaking
- Use cards eotj pictures
Editor's Notes
If we want assessment to be valid, reliable, and feasible, we need to specify:
What is assessed: according to the CEFR, communicative activities (contexts, texts, and tasks). See examples.
How performance is interpreted: assessment criteria. See examples
How to make comparisons between different tests and ways of assessment (for example, between public examinations and teacher assesment). Two main procedures:
Social moderation: discussion between experts
Benchmarking: comparison of samples in relation to standardized definitions and examples
Guidelines for good practice: EALTA
Types of tests:
Proficiency tests: designed to measure people’s ability in a language, regardless of any training. “Proficient”: command of the language, for a particular purpose or for general purposes.
Achievement tests: most teachers are not responsible for proficiency tests, but for achievement tests. They are normally related to language courses. Two approaches:
to base achievement tests on the textbook (or the syllabus), so that only what is covered in the classes is tested,
or, much better, to base test content on course objectives. More beneficial washback. The long-term interests of the students are best served by this approach.
Two types: final achievement tests, and progress achievement tests (formative assessment)
Diagnostic tests: Used to identify learners’ strengths and weaknesses (example: Dialang)
Placement tests: to place students at the stage most appropriate to their abilities
A test is valid if it measures accurately what it is intended to measure. Or, the information gained is an accurate representation of the proficiency of the candidate. This general type of validity is called “construct validity”, the validity of the construct, the thing we want to measure
Content validity: A test has it if its content constitutes a representative sample of the language skills or structures, etc. that it wants to measure. So, first, we need a specification of the skills of structures that we want to cover, and compare them with the test itself. For example, B2 writing skills, writing formal letters is one of the subskills shown in the specification, there are more, the more we cover, the more valid the test will be. The more content validity, the more construct validity and the more backwash effect.
Criterion-related validity: Results on the test agree with other (independent and highly dependable) results. This independent assessment is the criterion measure.
Two types:
Concurrent validity: we compare the criterion test and the test that we want to check. They both take place at about the same time.
Example 1: we administer a 45 m. oral test where all the subskills, tasks, operations, are tested. But only to a sample of the students. This is the criterion test. Then we do 10 m. interviews to the whole level of students. We compare the results, and they tell us whether 10 m. is enough or not. This is expressed in a “correlation coefficient” bw the criterion and the test being validated.
Example 2: we compare the results of a general test (Pruebas Estandarizadas) with teachers’ assessment.
Predictive validity: the test predicts future performance of the students. A placement test can easily be validated by the teachers teaching the students by checking if the students are well placed or not.
Validity in scoring: not only the items need to be valid, but also the way in which the responses are scored. For example, a reading test may call for short written responses. If the scoring of these responses takes into account spelling and grammar, then it is not valid (it is not measuring what it is intended to measure). Same for the scoring of writing or speaking.
Face validity: the test has to look as if it measures what it is supposed to measure. It is not a scientific notion, but it is important (for candidates, teachers, employers). For example, a written test to check pronunciation.
A test is valid if it measures accurately what it is intended to measure. Or, the information gained is an accurate representation of the proficiency of the candidate. This general type of validity is called “construct validity”, the validity of the construct, the thing we want to measure
Content validity: A test has it if its content constitutes a representative sample of the language skills or structures, etc. that it wants to measure. So, first, we need a specification of the skills of structures that we want to cover, and compare them with the test itself. For example, B2 writing skills, writing formal letters is one of the subskills shown in the specification, there are more, the more we cover, the more valid the test will be. The more content validity, the more construct validity and the more backwash effect.
Criterion-related validity: Results on the test agree with other (independent and highly dependable) results. This independent assessment is the criterion measure.
Two types:
Concurrent validity: we compare the criterion test and the test that we want to check. They both take place at about the same time.
Example 1: we administer a 45 m. oral test where all the subskills, tasks, operations, are tested. But only to a sample of the students. This is the criterion test. Then we do 10 m. interviews to the whole level of students. We compare the results, and they tell us whether 10 m. is enough or not. This is expressed in a “correlation coefficient” bw the criterion and the test being validated.
Example 2: we compare the results of a general test (Pruebas Estandarizadas) with teachers’ assessment.
Predictive validity: the test predicts future performance of the students. A placement test can easily be validated by the teachers teaching the students by checking if the students are well placed or not.
Validity in scoring: not only the items need to be valid, but also the way in which the responses are scored. For example, a reading test may call for short written responses. If the scoring of these responses takes into account spelling and grammar, then it is not valid (it is not measuring what it is intended to measure). Same for the scoring of writing or speaking.
Face validity: the test has to look as if it measures what it is supposed to measure. It is not a scientific notion, but it is important (for candidates, teachers, employers). For example, a written test to check pronunciation.
Reliability: A student being tested twice will get the same result (technical concept: the rank order of the candidates is replicated in two separate—real or simulated—administrations of the same assessment )
We compare two tests taken by the same group of students, and get a reliability coefficient: if all the students get exactly the same result, the coefficient is 1 (It never happens). High Stakes Tests need a higher coefficient than Lower Stakes exams. They shouldn’t depend on chance, or particular circumstances.
In order to get two comparable tests, there are two procedures:
Test-retest method: the students take the same test again
Alternate forms method: the students take two alternate forms of the same test
Split half method: you split the test into two (equivalent) halves and compare them as if they were two different tests. You get a “coefficient of internal consistency”.
We also need to know the standard error of measurement of a test. This is actually the opposite of the reliability coefficient and you can get it through statistical analysis. With this number, we can find out what the true score of a student is. For example, if we have a very reliable test, it will have a low standard error of measurement, and therefore, the student will always get a very similar result no matter how many times he takes the test. In a less reliable test, his true score would be less defined. The true score lies in a range that varies depending on the standard error of measurement of the test.
These numbers are important to compare tests and to take decisions (by companies, governments, etc.) based on those results.
Another statistical procedure commonly used now is Item Response Theory. Very technical.
Scorer reliability. There is also a scorer reliability coefficient, the level of agreement given by the same or different scorers on different occasions. If the scoring is not reliable, the rest results cannot be reliable.
Reliability: A student being tested twice will get the same result (technical concept: the rank order of the candidates is replicated in two separate—real or simulated—administrations of the same assessment )
We compare two tests taken by the same group of students, and get a reliability coefficient: if all the students get exactly the same result, the coefficient is 1 (It never happens). High Stakes Tests need a higher coefficient than Lower Stakes exams. They shouldn’t depend on chance, or particular circumstances.
In order to get two comparable tests, there are two procedures:
Test-retest method: the students take the same test again
Alternate forms method: the students take two alternate forms of the same test
Split half method: you split the test into two (equivalent) halves and compare them as if they were two different tests. You get a “coefficient of internal consistency”.
We also need to know the standard error of measurement of a test. This is actually the opposite of the reliability coefficient and you can get it through statistical analysis. With this number, we can find out what the true score of a student is. For example, if we have a very reliable test, it will have a low standard error of measurement, and therefore, the student will always get a very similar result no matter how many times he takes the test. In a less reliable test, his true score would be less defined. The true score lies in a range that varies depending on the standard error of measurement of the test.
These numbers are important to compare tests and to take decisions (by companies, governments, etc.) based on those results.
Another statistical procedure commonly used now is Item Response Theory. Very technical.
Scorer reliability. There is also a scorer reliability coefficient, the level of agreement given by the same or different scorers on different occasions. If the scoring is not reliable, the rest results cannot be reliable.
Item analysis:
Facility value
Discrimination indices: drop some, improve others
Analyse distractors
Item banking
SEE EXAMPLE FROM FUENSANTA
How to make tests more reliable (Hughes)
Take enough samples of behaviour. The more items, the more reliable. The higher stakes, the longer it should be. Example from the Bible. P. 45
Exclude items which do not descriminate well between weaker and stronger students
Do not allow candidates too much freedom. Example p. 46
Write unambiguous items: Critical scrutiny of colleagues, pre-testing (trialling, piloting)
Provide clear and explicit instructions: write them down, read them aloud. No problem with writing them in L1.
Ensure that tests are well laid out and perfectly legible
Make candidates familiar with format and testing techniques
Provide uniform and non-distracting conditions of administration (specified timing, good acoustic conditions)
Use items which permit scoring which is as objective as possible (better one-word response than multiple choice)
Make comparisons between candidates as direct as possible (no choice of items)
Provide a detailed scoring key
Train scorers
Agree acceptable responses and appropriate scores at the beginning of the scoring process. Score a sample. Choose representative examples. Agree. Then scorers can begin to score.
Identifty candidates by number not by name
Emply multiple, independent scorers. At least two, independently. Then, a third, senior scorer gets the results, and investigates discrepancies.
Washback/Backwash: (One of the) main reasons for a language teacher/school/department to use appropriate forms of assessment.
Test the abilities/skills you want to encourage. Give them sufficient weight in relation to other skills.
Sample widely and unpredictably: Test across the full range of the specifications
Use direct testing
Make testing criterion-referenced (CEFR)
Base achievement tests on objectives
Ensure that the test is known and understood by students and teachers (the more transparent, the better)
(Where necessary, provide assistance to teachers)
Counting the cost: Individual direct testing is expensive, but what is the cost of not achieving beneficial washback
Calibrate scales: collect samples of performance, and use them as models, reference points (European Study)
Common Test Techniques
We need techniques which:
- will elicit behaviour which is a reliable and valid indicator of the ability in which we are interested;
- will elicit behaviour which can be reliably scored;
- are as economical of time and effort as possible;
will have a beneficial backwash effect, where this is relevant.
MULTIPLE CHOICE
Advantages:
Reliable
Economical
Good for receptive skills
(It used to be as the perfect, almost only way to test)
Disadvantages:
Only for recognition
Guessing may have a considerable bu unknowable effect
The technique severely restricts what can be tested
It is very difficult to write successful items
Washback may be harmful
Cheating may be facilitated
YES/NO TRUE/FALSE ITEMS
Essentially multiple choice, but with a 50 % chance4 of getting it right. Ok in class activities. Not appropriate in real testing.
SHORT-ANSWER ITEMS
Advantages.
Less guessing
No need for distractors
Less cheating
Items are easier to write
Disadvantages
Responses may take longer
The test taker has to produce language (mixture of skills in a receptive test) (TRY TO MAKE RESPONSES REALLY SHORT)
Judging may be required (less validity or reliability)
Scoring may take longer (SOLUTIONS: MAKE THE REQUIRED RESPONSE UNIQUE)
GAP FILLING ITEMS very similar to short-answer items
Set representative tasks
Specify all possible content (in the specifications)
Include a representative sample of the specified content (in the test)
Elicit valid samples of writing ability
Set as many separate tasks as feasible
Test only writing ability and nothing else (creativity, imagination, etc. No extra long instructions with complicated reading)
Restrict candidates
Ensure valid and reliable scoring:
Set as many tasks as possible
Restrict candidates
Give no choice of tasks
Ensure long enough samples
Create appropriate scales for scoring: HOLISTIC/ANALYTIC See examples. HOLISTIC. Good if many scorers. ANALYTIC: equal or unequal weight to the different parts, main disadvantage: time-consuming, if too much attention is payed to the parts, one may forget the general impression. IMPORTANT POTENTIAL FOR WASHBACK.
Calibrate the scale to be used (collect samples. Choose representative ones. Use them as reference points. This is called “benchmarking”)
Select and train scorers
Follow acceptable scoring procedures: benchmarking, two scorers (and a third, senior one for discrepancies), carry out statistical analysis
“The most highly prized language skill”, a source of cultural capital, Lado’s Language Testing (1961). However, it hasn’t always been properly assessed.
Challenges: ephemeral, intangible. Solutions: recording it, and also sound waves, spectrographs
Some tests (TOEFL in particular) have a long history of ignoring it: Only in 2005 TOEFL iBT/Contrast with Cambridge Certificate of Proficiency in English (1913) which already included it. However, Grammar-Translation approaches ignored it almost completely. Kaulfers 1944 created the first scales used to assess oral proficiency, designed for the military abroad
Key notion: not accent, but intelligibility (the ease or difficulty with which a listener understands L2 speech. You can be highly intelligible with a non-native accent. It is only when the accent interferes with a learner’s ability that it should be considered in speaking scales.
Very different approaches.
Indirect (multiple choice as an indicator, not really valid or reliable)
Direct or Semi-direct (responding to stimulus from a computer, TOEFL ibt, OTE, Aptis). Problems: raters and rating scales (which oversimplify the complexity of oral speech). Despite the practical challenges, they are the only valid formats for assessing L2 speech today. Conflict with the American tradition of “psychometrically influenced assessment tradition” focusing on the technical (statistical) reliability of test items (multiple choice) and the most administratively feasible test formats and item types in the context of large-scale, high-stakes tests (GRE?)
The future?: Fully automated L2 speaking tests: Versant test, Speechrater. Automatic scoring systems (measuring grammatical accuracy, lexical frequency, acoustic variables, temporal variables)
Not only speaking, also interaction (listening and speaking): Cambridge included interaction in 1996. Washback effect (usual practice in class, pairwork, groupwork). Problems: peer interlocutor variables (L2 proficiency, L1 background, gender, personality, etc). Solutions: more tasks.
Set representative tasks
Specify all possible content
Include a representative sample of the specified content
Elicit valid samples of oral ability.
Techniques:
Interview (the candidate may feel intimidated): Questions, pictures, role play, interpreting (L1 to L2), prepared monologue, reading aloud
Interaction with fellow candidates: discussion, roleplay
Responses to audio- or video-recordings (semi-direct)
Plan and structure the test carefully
Make the oral test as long as it is feasible
Plan the test carefully
As many tasks (“fresh starts”) as possible
Use a second tester
Set only tasks that candidates could do easily in L1
Quiet room with good acoustics
Put candidates at ease (at first, easy questions, not assessed, problem with note-taking?)
Collect enough relevant information
Do not talk too much
(select interviewers carefully and train them)
Ensure valid and reliable scoring:
Create appropriate scales for scoring: HOLISTIC/ANALYTIC. Used as a check on each other
Calibrate the scale to be used
Select and train scorers (different from interviewers if possible)
Follow acceptable scoring procedures
PROBLEMS:
Indirect assessment: the exercise of receptive skills does not manifest itself directly. We need an instrument.
We read in very different ways: scanning, skimming, inferring, intensive, extensive reading… All of them should be specified and tested
SOME TIPS
As many texts and operations as possible (Dialang). (Time limits for scanning or skimming?)
Avoid texts which deal with general knowledge (answers will be guessed)
Avoid disturbing topics, or texts students might have read
Use, as much as possible, authentic texts
Techniques: better short answer and gap filling than multiple choice. Also information transfer.
Task difficulty can be lower than text difficulty
Items should follow the order of the text
Make items independent of each other
Do not take into account errors of grammar or spelling.
Similar PROBLEMS to listening:
Indirect assessment: the exercise of receptive skills does not manifest itself directly. We need an instrument.
We listen in very different ways: scanning, skimming, inferring, intensive, extensive listening… All of them should be specified and tested
And to Speaking:
Transient nature of speech
http://www.usingenglish.com/articles/why-your-students-have-problems-with-listening-comprehension.html
Similar tips from Reading (go back to the list)
If recording is used, make it as natural as possible (with typical spoken redundancy). Don’t read aloud written texts.
Items should be far apart in the text (to have time to write them down)
Give students time to become familiar with the tasks
Techniques: apart from multiple choice, shot answers and gap filling, information transfer (draw a map of the accident), note taking, partial dictation (problem: do you consider spelling?), transcription (spelling names, numbers: real life task)
Moderation (more teachers, trialing) is essential
How many times? Why two? Never three
GRAMMAR:
Why? Easy to test, Content validity: more than in any of the skills (Skills: we just cover a few of the topics, or operations from the specifications. Grammar: we can cover many more items)
Why not? Harmful washback effect
Maybe not in proficiency tests, but, if grammar is taught (and it almost always is), it should be included in achievement tests, placement and diagnostic tests. However, because of the potential harmful washback effect, it should not be given too much (porcentual) prominence.
Specifications: from the Council of Europe books (Threshold, etc.)
Techniques: Gap filling, rephrasings, completion
Don’t penalize for mistakes that were not tested (-s if the item is testing relatives, for example)
VOCABULARY
Why (not)? Similar arguments as for grammar.
Specifications: use frequency considerations (cobuild dictionaries)
Techniques:
Recognition: Recognise synonims, recognise definitions, recognise appropriate word for context
Production: pictures, definitions, gap filling,
Special techniques which are more useful in tests where washback is not important: placement tests, for example
Types:
Cloze test (from closure). Based on the idea of “reduced redundancy”. Texts are always redundant. If we reduce the redundancy (by deleting a few words), native speakers are easily able to cope and guess the missing words. Originally, every seventh word. In the 80s ot ised to be considered as a language testing panacea (panasía). Easy to construct, administer and score. Unfortunately, poor validity. Native speakers cannot always guess the words. SUBTYPES:
Selected deletion cloze
Conversational cloze
The C-Test: a variety of cloze, with the second half of every second word deleted. Puzzle-like
Dictation: traditionally used (particularly in places like France, but not only). However, in the 60s, dictation testing was considered misguided. Later, nevertheless, research showed correlation between scores on dictation tests and scores on more complex tests, or on cloze tests. They are easy to create, and easy to adminster, but very difficult to score properly.
Main problem with all of these tests: horrible washback effect.
Primary School: Other types of assessment are more appropriate. However, a common yardstick at the end is necessary: Pruebas Estandarizadas.
Good opportunity to develop good attitudes towards assessment. Recommendations:
Make testing an integral part of assessment, and assessment an integral part of the teaching program
Feedback from tests should be immediate and positive
Self assessment should be part of the teaching program
Washback is more important than ever
TIPS
Short tasks: Short attention span
Use stories and games
Use pictures and color
Don’t forget that children are still developing L1 and cognitive abilities
Include interaction
SOME TECHNIQUES:
Placing objects or identifying people
Multiple choice pictures
Colour and draw
Use pictures in reading and in writing
Cartoon stories for writing
Long warm-ups in speaking
Use cards and pictures