Your SlideShare is downloading. ×
ILR 101 January 23, 2009 Language Testing and Assessment Unit, FBI Maria M. Brau and Rachel L. Brooks
Topics Covered <ul><li>ILR background  </li></ul><ul><li>ILR scale </li></ul><ul><li>Language skills  </li></ul><ul><li>IL...
Interagency Language Roundtable (ILR) <ul><li>Organization of representatives from different government agencies that prov...
ILR Skill Level Descriptions <ul><li>Descriptions of the minimum language requirements a person must have in order to perf...
ILR Scale <ul><li>Ordinal scale ranging from 0 to 5 </li></ul><ul><ul><li>Interval scale: distances between levels are equ...
Typical Interval Scales Poor Excellent Good Fair Satisfactory 100 0 50 A B C D F Surveys: Academic:
0 1 2 3 4 5 Level 3 Range ILR Skill Level Descriptions
Cooking Scale <ul><li>0: No cooking ability </li></ul><ul><li>Combine foods. </li></ul><ul><li>1: Prepare </li></ul><ul><l...
Hiking Scale <ul><li>0: Unable to hike, even on paved roads. </li></ul><ul><li>1: Able to hike on flat, paved paths. </li>...
ILR Scale 1 2 3 4 5 Moving upward is increasingly difficult.
5 4 0 1 2 3 Level 5 Level 4 Level 3 Level 2 Level 1 Level 0 Level 3 Range 3 ILR Skill Levels Each level represents a range.
e, t e.t.
e, t, a e.t. ate eat tea at ta
e, t, a, m e.t. ate eat tea at ta mate meat meta tame team mat met tam am me ma
e, t, a,  m, s e.t. ate eat tea at ta mate meat meta tame team mat met tam am me ma mates meats steam tames teams east eat...
ILR Skill Level Descriptions <ul><li>Speaking </li></ul><ul><li>Listening </li></ul><ul><li>Reading  </li></ul><ul><li>Wri...
Speaking 3  (General Professional Proficiency)   <ul><li>Able to speak the language with sufficient structural accuracy an...
ILR Levels 2, 2+, 3 Level Speaking Listening Reading Writing 2 Able to satisfy routine social demands and limited work req...
ILR Level 2 Speaking Able to satisfy routine social demands and limited work requirements. Listening Sufficient comprehens...
ILR Level 2+ Speaking Able to satisfy most work requirements with language usage that is often, but not always, acceptable...
ILR Level 3 Speaking Able to speak the language with sufficient structural accuracy and vocabulary to participate effectiv...
“ Knowing” a language <ul><li>Language skills are  separate . </li></ul><ul><ul><li>A person can have different proficienc...
Four Primary Skills Translation Interpretation <ul><li>Reading </li></ul><ul><li>Receptive </li></ul><ul><li>Delayed </li>...
Types of Listening Interactive Listening (Participatory) Monitoring (Recorded Speech) Monitoring (Live Speech) Static List...
Translation Listening: FL Writing: Eng Listening: Eng Writing: FL Monitoring (Audio Translation) Prerequisite Skills Readi...
Interpretation Listening: FL Speaking: Eng Listening: Eng Speaking: FL Consecutive & Simultaneous Interpretation Prerequis...
Transcription (Monolingual) Listening: FL Writing: FL Listening: Eng Writing: Eng Transcription Prerequisite Skills
Capacity to Translate 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 NSL LSL Ideal Normal Reading Comprehension of the Source Language Wr...
From theory to practice <ul><li>Hypothesis 1 :  </li></ul><ul><li>If you can read a foreign language, you can translate. <...
Post 9/11 Arabic Applicants 1 10 12 (0.8%) Failed Reading/Passed TT 221 634 781 (54.3%) Passed Reading/Failed TT 369 875 1...
Percent Concordance: Final Derived VTE and EWT Score   Italian Overall (%) Vietnamese Overall (%) Turkish Overall (%) Aver...
Practice Translation:  English to English Paraphrase the following sentences <ul><li>I’m going to throw out my sandwich. <...
Maria M. Brau [email_address] Rachel L. Brooks [email_address]
Upcoming SlideShare
Loading in...5
×

ILR 101

1,994

Published on

This is a brief by Maria M. Brau and Rachel L. Brooks and the FBI but I found it extremely beneficial in understanding the ILR scale and DoD relationship. It also points out the challenges of trying to quantify something rather qualitative in nature--language!

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,994
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • 01/13/10
  • 01/13/10 Topics marked are those covered in the ILR 101 basic slide set, presented at the January 2009 Plenary. Other topics are added, depending on participant needs.
  • 01/13/10
  • 01/13/10
  • 01/13/10 Purpose: introduce participants to the ILR scale by noting it is different from the interval scales with which they are familiar.
  • 01/13/10 Purpose: to provide examples of familiar interval scales. The first is a Likert scale, used in consumer surveys. The second reflects the grading scale normally used in American educational institutions. Here, a score reflects one particular point on the scale. Only scores in the upper range are considered “good” or “acceptable.”
  • 01/13/10 Purpose: to provide examples of familiar interval scales. The first is a Likert scale, used in consumer surveys. The second reflects the grading scale normally used in American educational institutions. Here, a score reflects one particular point on the scale. Only scores in the upper range are considered “good” or “acceptable.”
  • 01/13/10 Purpose: to illustrate the principles behind the ILR scale by applying it to a familiar activity, such as cooking. Asking the participants to engage in this practical exercise helps them understand how acquisition of additional skills expands an individual’s abilities. The exercise also brings out the traditional distinctions between levels: a Level 1 can survive, a Level 2 has enough skills to satisfy daily needs, (a 2+ is the level at which individuals will normally resolve daily challenges successfully), a Level 3 is capable of producing professional results, a Level 4 is a chef using complex techniques that produce consistent results, and a Level 5 is recognized as a master chef.
  • 01/13/10 Purpose: to illustrate the principles behind the ILR scale by applying it to a familiar activity, such as hiking. Again, below level 3, the hiker operates in a mostly familiar setting, without too many complications. The hiker above Level 3 requires specialized skills and equipment, and is able to lead.
  • 01/13/10 Purpose: to illustrate graphically that achieving a higher score is progressively more difficult, since it involves not only increased vocabulary and structural control, but also linguistic competence to handle tasks at the upper levels.
  • 01/13/10 Purpose: to show the inverted pyramid concept, traditionally used by ILR tester-trainers to illustrate ranges within levels.
  • 01/13/10 Purpose: to show how additions multiply capacity. With 2 letters, 1 acronym is possible but 0 words.
  • 01/13/10 Add 1 letter, and 5 words are possible.
  • 01/13/10 Another letter jumps the number of words to 16.
  • 01/13/10 One more letter allows for the formation of 36 words.
  • 01/13/10
  • 01/13/10 Purpose: to show the structure of an ILR skill level description. A bolded statement characterizes general ability at that level, followed by several statements providing details. There is also an examples section that is not shown in this slide.
  • 01/13/10 Purpose: to show how similar concepts are conveyed across the proficiency skills.
  • 01/13/10 Purpose: to show that work should not be assigned to a person based on a test score in a single skill.
  • 01/13/10 Purpose: to show that (1) each of the proficiency skills has different attributes, and (2) each of the combined skills has different prerequisites. Therefore, it does not necessarily follow that a good translator can be a good interpreter, and vice versa. Translators use delayed skills, are able to craft a translation over time, with revisions and edits. Interpretation requires immediate language processing, involving the use of memory.
  • 01/13/10 Purpose: to show that there are differences in a single skill, e.g., listening. In interactive listening, the person is a participant in the conversation and can request repetitions and clarifications. A non-participant can replay speech only if the source is a recording but can never request clarification.
  • 01/13/10 Purpose: to show that the prerequisites for translation differ depending on the task.
  • 01/13/10 Purpose: to show that the prerequisites for interpretation differ depending on the task. Note that those for audio translation (listening and writing) and those for sight translation (reading and speaking) are exact opposites.
  • 01/13/10 Purpose: to show that since the prerequisite skills for transcription are monolingual, the task does not need to be assigned to a translator and much less to an interpreter, whose prerequisite skills do not include writing.
  • 01/13/10 Purpose: to show that a person’s capacity to translate is limited by the lower prerequisite skill. A person who is a native of the source language (NSL) may be highly proficient in reading the source document, but a weakness in writing the target language will limit translation capacity. A person who is a learner of the source language (LSL) may have a lower reading comprehension and a higher writing ability in the target language. In both cases, the person will not perform higher than the lower prerequisite skill. It must be noted that prerequisites limit maximum capacity, but lack of congruity judgment (the ability to choose proper equivalents) may further bring down the quality of the product. The effect of congruity judgment is not shown in the chart.
  • 01/13/10 Purpose: to show that FBI research tends to confirm that a reading test, a writing test, or a combination of the two, should not be used to replace a translation test. These are testing instruments that focus on assessing congruity judgment.
  • 01/13/10 The study investigated the relationship between reading ability and translation ability. Examinees took the Defense Language Proficiency Test (DLPT) for both Reading and Listening in Arabic and the Arabic Translation Test. Passing rates for all tests were set at 2+. Although almost 75% of applicants passed the DLPT for Reading, only 20% of applicants passed the Translation Test, meaning that more than 50% of those who passed the Reading DLPT, failed the Translation Test.
  • 01/13/10 The study investigated the relationship between writing ability and translation ability. Participants were natives of the foreign language, and could read the source text material. Both English Writing Tests (EWTs) and Verbatim Translation Exams (VTEs), from the foreign language into English, were administered to participants from three different language groups. On average, only 9% of the participants’ scores matched on the two tests, meaning 9% met their maximum capacity to translate. Approximately 35% had scores in the same level/plus level range. About 33% of participants had one level difference between the two tests, and each time the VTE score was lower than the EWT score. In almost 32% of the cases, the VTE score was two main ILR levels lower than the EWT score.
  • 01/13/10 Purpose: to provide an opportunity for those who are not translators to experience what translation is like. The task was to paraphrase the sentence given with another sentence in English, making sure not to use any of the words in the original sentences. Two possible responses are given for the first two sentences.
  • Transcript of "ILR 101"

    1. 1. ILR 101 January 23, 2009 Language Testing and Assessment Unit, FBI Maria M. Brau and Rachel L. Brooks
    2. 2. Topics Covered <ul><li>ILR background </li></ul><ul><li>ILR scale </li></ul><ul><li>Language skills </li></ul><ul><li>ILR Skill Level Descriptions </li></ul><ul><li>In-house testing studies </li></ul><ul><li>Native speakers/heritage speakers/learners </li></ul><ul><li>Translation text difficulty levels </li></ul><ul><li>Audio text difficulty levels </li></ul><ul><li>ILR-based translation exams </li></ul><ul><li>Tests administered </li></ul><ul><li>Testing policies and procedures </li></ul>
    3. 3. Interagency Language Roundtable (ILR) <ul><li>Organization of representatives from different government agencies that provides language services along with other language professionals </li></ul><ul><li>Goals </li></ul><ul><ul><li>Standardization of testing criteria </li></ul></ul><ul><ul><li>Collaboration of training, testing, and translation methodology between agencies </li></ul></ul><ul><ul><li>Dissemination of new language information and practices from both inside and outside the government arena </li></ul></ul>
    4. 4. ILR Skill Level Descriptions <ul><li>Descriptions of the minimum language requirements a person must have in order to perform certain tasks in a skill. </li></ul><ul><li>Skills measured: </li></ul><ul><ul><li>Listening </li></ul></ul><ul><ul><li>Reading </li></ul></ul><ul><ul><li>Speaking </li></ul></ul><ul><ul><li>Writing </li></ul></ul><ul><ul><li>Translation </li></ul></ul><ul><ul><li>Interpretation </li></ul></ul><ul><li>Used as a standard for all US Government agencies </li></ul><ul><li>Scale ranges from 0 to 5, with plus levels </li></ul>
    5. 5. ILR Scale <ul><li>Ordinal scale ranging from 0 to 5 </li></ul><ul><ul><li>Interval scale: distances between levels are equidistant </li></ul></ul><ul><ul><li>Ordinal scale: distances between levels increase as levels increase </li></ul></ul>
    6. 6. Typical Interval Scales Poor Excellent Good Fair Satisfactory 100 0 50 A B C D F Surveys: Academic:
    7. 7. 0 1 2 3 4 5 Level 3 Range ILR Skill Level Descriptions
    8. 8. Cooking Scale <ul><li>0: No cooking ability </li></ul><ul><li>Combine foods. </li></ul><ul><li>1: Prepare </li></ul><ul><li>Heat liquid or food for a certain time, to a certain temperature. </li></ul><ul><li>2: Follow recipes </li></ul><ul><li>Combine ingredients, follow instructions. </li></ul><ul><li>3: Catering for small groups </li></ul><ul><li>Manage many types of recipes, arriving at different times. </li></ul><ul><li>4: Chef </li></ul><ul><li>Uses advanced techniques to prepare complex menus. </li></ul><ul><li>5: Master Chef </li></ul><ul><li>Products: </li></ul><ul><li>Bread and butter </li></ul><ul><li>Cereal </li></ul><ul><li>Salad </li></ul><ul><li>Products: </li></ul><ul><li>Tea </li></ul><ul><li>Boil spaghetti, add sauce and cheese </li></ul><ul><li>Products: </li></ul><ul><li>Hamburger </li></ul><ul><li>Casserole </li></ul><ul><li>Products: </li></ul><ul><li>Lobster bisque </li></ul><ul><li>Grilled chicken stuffed with spinach, roasted potatoes, and steamed zucchini served with lemon sauce. </li></ul><ul><li>Flan </li></ul><ul><li>Products: </li></ul><ul><li>Andre Terrail’s pike dumplings </li></ul><ul><li>Poached foie gras with Pinot gris wine, turnip leaves, and dried beetroot </li></ul><ul><li>Marinated Gulf shrimp with Spanish paprika and toasted Marcona almonds, pink grapefruit emulsion </li></ul><ul><li>Braised lobster with sea-urchin sauce, velvety coral lentils, olive pulp in aspic </li></ul><ul><li>Whipped mild cocoa cream with saffron </li></ul>
    9. 9. Hiking Scale <ul><li>0: Unable to hike, even on paved roads. </li></ul><ul><li>1: Able to hike on flat, paved paths. </li></ul><ul><li>2: Able to complete most regional hikes, staying mostly on course, despite minor geographic obstacles. </li></ul><ul><li>3: Able to complete multi-day hikes that may involve unmarked trails or rigorous topographical changes in a variety of climates. </li></ul><ul><li>4: Able to accomplish treks across traverse terrain that require specialized equipment and skills. May serve as a guide. </li></ul><ul><li>5: Able to successfully summit any peak or complete any course on all attempts. </li></ul>
    10. 10. ILR Scale 1 2 3 4 5 Moving upward is increasingly difficult.
    11. 11. 5 4 0 1 2 3 Level 5 Level 4 Level 3 Level 2 Level 1 Level 0 Level 3 Range 3 ILR Skill Levels Each level represents a range.
    12. 12. e, t e.t.
    13. 13. e, t, a e.t. ate eat tea at ta
    14. 14. e, t, a, m e.t. ate eat tea at ta mate meat meta tame team mat met tam am me ma
    15. 15. e, t, a, m, s e.t. ate eat tea at ta mate meat meta tame team mat met tam am me ma mates meats steam tames teams east eats mast mats mesa same sate seam seat tams stem teas sat sea as
    16. 16. ILR Skill Level Descriptions <ul><li>Speaking </li></ul><ul><li>Listening </li></ul><ul><li>Reading </li></ul><ul><li>Writing </li></ul><ul><li>Translation </li></ul><ul><li>Interpretation </li></ul><ul><li>For full texts, go to: </li></ul><ul><ul><li>www.govtilr.org </li></ul></ul>Proficiency (Single Skills) Performance (Combined Skills)
    17. 17. Speaking 3 (General Professional Proficiency) <ul><li>Able to speak the language with sufficient structural accuracy and vocabulary to participate effectively in most formal and informal conversations on practical, social, and professional topics. Nevertheless, the individual's limitations generally restrict the professional contexts of language use to matters of shared knowledge and/or international convention. Discourse is cohesive. The individual uses the language acceptably, but with some noticeable imperfections; yet, errors virtually never interfere with understanding and rarely disturb the native speaker. The individual can effectively combine structure and vocabulary to convey his/her meaning accurately. The individual speaks readily and fills pauses suitably. In face-to-face conversation with natives speaking the standard dialect at a normal rate of speech, comprehension is quite complete. Although cultural references, proverbs, and the implications of nuances and idiom may not be fully understood, the individual can easily repair the conversation. Pronunciation may be obviously foreign. Individual sounds are accurate; but stress, intonation, and pitch control may be faulty. </li></ul>Able to speak the language with sufficient structural accuracy and vocabulary to participate effectively in most formal and informal conversations on practical, social, and professional topics. The individual can effectively combine structure and vocabulary to convey his/her meaning accurately.
    18. 18. ILR Levels 2, 2+, 3 Level Speaking Listening Reading Writing 2 Able to satisfy routine social demands and limited work requirements. Sufficient comprehension to understand conversations on routine social demands and limited job requirements. Sufficient comprehension to read simple, authentic written material in a form equivalent to usual printing or typescript on subjects within a familiar context. Able to write routine social correspondence and prepare documentary materials required for most limited work requirements. 2+ Able to satisfy most work requirements with language usage that is often, but not always, acceptable and effective. Sufficient comprehension to understand most routine social demands and most conversations on work requirements as well as some discussions on concrete topics related to particular interests and special fields of competence. Sufficient comprehension to understand most factual material in non-technical prose as well as some discussions on concrete topics related to special professional interests. Shows ability to write with some precision and in some detail about most common topics. 3 Able to speak the language with sufficient structural accuracy and vocabulary to participate effectively in most formal and informal conversations on practical, social, and professional topics. Able to understand the essentials of all speech in a standard dialect including technical discussions within a special field. Able to read within a normal range of speed and with almost complete comprehension of a variety of authentic prose material on unfamiliar subjects. Able to use the language effectively in most formal and informal written exchanges on practical social and professional topics.
    19. 19. ILR Level 2 Speaking Able to satisfy routine social demands and limited work requirements. Listening Sufficient comprehension to understand conversations on routine social demands and limited job requirements. Reading Sufficient comprehension to read simple, authentic written material in a form equivalent to usual printing or typescript on subjects within a familiar context. Writing Able to write routine social correspondence and prepare documentary materials required for most limited work requirements.
    20. 20. ILR Level 2+ Speaking Able to satisfy most work requirements with language usage that is often, but not always, acceptable and effective. Listening Sufficient comprehension to understand most routine social demands and most conversations on work requirements as well as some discussions on concrete topics related to particular interests and special fields of competence. Reading Sufficient comprehension to understand most factual material in non-technical prose as well as some discussions on concrete topics related to special professional interests. Writing Shows ability to write with some precision and in some detail about most common topics.
    21. 21. ILR Level 3 Speaking Able to speak the language with sufficient structural accuracy and vocabulary to participate effectively in most formal and informal conversations on practical, social, and professional topics. Listening Able to understand the essentials of all speech in a standard dialect including technical discussions within a special field. Reading Able to read within a normal range of speed and with almost complete comprehension of a variety of authentic prose material on unfamiliar subjects. Writing Able to use the language effectively in most formal and informal written exchanges on practical social and professional topics.
    22. 22. “ Knowing” a language <ul><li>Language skills are separate . </li></ul><ul><ul><li>A person can have different proficiency levels in different skills. </li></ul></ul><ul><ul><ul><li>Speaking, Listening, Reading, Writing </li></ul></ul></ul><ul><li>Language skills are combined . </li></ul><ul><ul><li>A person must combine various skills in order to perform language tasks. </li></ul></ul><ul><ul><ul><li>Holding a conversation, translating a document, monitoring a phone call, interpreting </li></ul></ul></ul>
    23. 23. Four Primary Skills Translation Interpretation <ul><li>Reading </li></ul><ul><li>Receptive </li></ul><ul><li>Delayed </li></ul><ul><li>Listening </li></ul><ul><li>Receptive </li></ul><ul><li>Immediate </li></ul><ul><li>Writing </li></ul><ul><li>Productive </li></ul><ul><li>Delayed </li></ul><ul><li>Speaking </li></ul><ul><li>Productive </li></ul><ul><li>Immediate </li></ul>
    24. 24. Types of Listening Interactive Listening (Participatory) Monitoring (Recorded Speech) Monitoring (Live Speech) Static Listening (Non-participatory) No opportunity to clarify Can request clarification
    25. 25. Translation Listening: FL Writing: Eng Listening: Eng Writing: FL Monitoring (Audio Translation) Prerequisite Skills Reading: FL Writing: Eng Reading: Eng Writing: FL Document Translation
    26. 26. Interpretation Listening: FL Speaking: Eng Listening: Eng Speaking: FL Consecutive & Simultaneous Interpretation Prerequisite Skills Reading: FL Speaking: Eng Reading: Eng Speaking: FL Sight Translation
    27. 27. Transcription (Monolingual) Listening: FL Writing: FL Listening: Eng Writing: Eng Transcription Prerequisite Skills
    28. 28. Capacity to Translate 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 NSL LSL Ideal Normal Reading Comprehension of the Source Language Writing Ability in the Target Language Maximum Translation Ability
    29. 29. From theory to practice <ul><li>Hypothesis 1 : </li></ul><ul><li>If you can read a foreign language, you can translate. </li></ul><ul><li>True or False </li></ul><ul><li>Hypothesis 2 : </li></ul><ul><li>If the foreign language is your native language and you are proficient in English, you can translate. </li></ul><ul><li>True or False </li></ul>Testing one prerequisite skill is not sufficient to predict translation ability.
    30. 30. Post 9/11 Arabic Applicants 1 10 12 (0.8%) Failed Reading/Passed TT 221 634 781 (54.3%) Passed Reading/Failed TT 369 875 1149 (79.9%) Failed Translation Test 113 202 289 (20.1%) Passed Translation Test 151 243 371 (25.8%) Failed Reading (DLPT) 331 834 1067 (74.2%) Passed Reading (DLPT) 253 563 683 (47.5%) Failed Listening (DLPT) 229 514 755 (52.5%) Passed Listening (DLPT) 482 1077 1438 Total Applicants L1: English L1: Arabic Total  
    31. 31. Percent Concordance: Final Derived VTE and EWT Score   Italian Overall (%) Vietnamese Overall (%) Turkish Overall (%) Average Overall (%) Exact Match 3.75 19.82 2.50 8.69 Within Level Match 28.75 50.16 26.30 35.07 One Level Difference (VTE higher or lower) 36.25 32.39 31.30 33.28 Greater than One Level Difference (VTE higher) 0.00 0.00 0.00 0.00 Greater than One Level Difference (VTE lower) 35.00 17.45 42.50 31.65
    32. 32. Practice Translation: English to English Paraphrase the following sentences <ul><li>I’m going to throw out my sandwich. </li></ul><ul><li>The child toddled around the den. </li></ul><ul><li>The incoming director of national intelligence plans to change security rules to make it easier for intelligence agencies to hire first-generation Arab Americans for highly sensitive jobs. </li></ul><ul><li>The defendant, who has worked at the embassy for two years, has a wife and children living abroad, said the embassy's office manager. </li></ul><ul><li>At the time of the apparent abduction, which was first reported by the local paper, the accused was traveling with two colleagues, officials said. </li></ul><ul><li>The sooner, the better. </li></ul>A certain pre-adolescent human moved in an awkward, unbalanced manner about that particular room purposed for recreation. The person with the intention to dispose of her bread-enveloped meal is me.
    33. 33. Maria M. Brau [email_address] Rachel L. Brooks [email_address]

    ×