DO INDONESIANS MAKE MOBILE CALLS AND DO RUSSIANS GO ON HOLIDAY? NHTV Breda – january 2008 Katholieke Universiteit Leuven Universiteit van Amsterdam E.Teijsse-Jongkind @uva.nl [email_address] The effect of culture on testing The Certificate Dutch as a Foreign Language

Content of this presentation Purpose of this presentation: to be aware of cultural bias in the classroom Background of the CNaVT (Certificaat Nederlands als Vreemde Taal) Definition of cultural bias and the effect of culture on different levels Task: Does this test contain cultural bias? How does the CNaVT manage the effect of culture in the CNaVT-exams? Discussion: own experiences with cultural bias

Purpose of this presentation: to be aware of cultural bias in the classroom

Background of the CNaVT (Certificaat Nederlands als Vreemde Taal)

Definition of cultural bias and the effect of culture on different levels

Task: Does this test contain cultural bias?

How does the CNaVT manage the effect of culture in the CNaVT-exams?

Discussion: own experiences with cultural bias

The Certificate of Dutch as a Foreign Language CNaVT (Certificaat Nederlands als Vreemde Taal) administration in more than 40 countries around the world: from Madrid to Jakarta and from Warsaw to Rio de Janeiro 2500 candidates each year

CNaVT (Certificaat Nederlands als Vreemde Taal)

administration in more than 40 countries around the world: from Madrid to Jakarta and from Warsaw to Rio de Janeiro

2500 candidates each year

Assess as direct as possible Based on a needs analysis authenticity: 6 exams represent 6 different domains task-based: exam tasks represent real life language use situations

Based on a needs analysis

authenticity: 6 exams represent 6 different domains

task-based: exam tasks represent real life language use situations

The effect of culture: cultural bias Definition of cultural bias: Scores of groups are distorted, because the items favour some groups over others. This distortion is not caused by a difference in (language) proficiency.

Definition of cultural bias:

Scores of groups are distorted, because the items favour some groups over others. This distortion is not caused by a difference in (language) proficiency.

THE EFFECT OF CULTURE ON DIFFERENT LEVELS (Van Baalen, Beheydt & van Kalsbeek, 2003) On the level of: 1. Words and expressions 2. Images 3. Subjects and concepts Knowledge of the world Relation examiner - candidate Classroom / test situation

(Van Baalen, Beheydt & van Kalsbeek, 2003)

On the level of:

1. Words and expressions

2. Images

3. Subjects and concepts

Knowledge of the world

Relation examiner - candidate

Classroom / test situation

1. WORDS and EXPRESSIONS  verpotteren  flitspaalgedrag speeddaten  (zit)bank vs. zetel  lopen vs. stappen  namiddag  iets afkloppen/ hout vasthouden

 verpotteren

 flitspaalgedrag

speeddaten

 (zit)bank vs. zetel

 lopen vs. stappen

 namiddag

 iets afkloppen/ hout vasthouden

2. IMAGES

3. SUBJECTS

4. KNOWLEDGE OF THE WORLD example 1 You want to organize a party for your birthday. Describe what you want to your friend.

You want to organize a party for your birthday.

Describe what you want to your friend.

4. KNOWLEDGE OF THE WORLD example 2 vs.

5. RELATION EXAMINER-STUDENT

6. CLASSROOM PRACTICE/ TEST SITUATION

MINI TEST Read the test and answer the following question. Does this test contain cultural bias for some groups of students? Which levels of cultural bias can appear? Why?

Read the test and answer the following question.

Does this test contain cultural bias for some groups of students? Which levels of cultural bias can appear? Why?

A TENSION Validity in the sense that we want our exams to measure language proficiency. Reliability in the sense that we want to be able to state that our exams are not influenced by other factors like the culture of the candidate.

Validity in the sense that we want our exams to measure language proficiency.

DIF ANALYSIS DIF Differential Item Functioning = the statistical view of cultural bias

CNaVT item bias research Example 1 Country: Indonesia Item: Question in form ‘ What kind of object have you lost?’

CNaVT item bias research Example 2 Country: Russia Item: Question from friend in letter (…) ‘But I often think back with nostalgia about my holiday in your country. (…) And you? What did you do during your holidays? Did you travel? Where to? Or did you stay at home and what did you do?’ (…)

How does the CNaVT manage the effect of culture? awareness of what we test: what (concerning culture) is relevant, what is irrelevant generate input on possible cultural bias from different sources in different stadia In case of suspicion of irrelevance: DIF analysis

awareness of what we test: what (concerning culture) is relevant, what is irrelevant

generate input on possible cultural bias from different sources in different stadia

In case of suspicion of irrelevance: DIF analysis

BUT… CNaVT is a test of Dutch as a foreign language! Difference between test situation and the classroom Cultural bias is not something that has to be avoided in the classroom But: teachers should use cultural miscommunications to learn from it Intercultural competence! Teacher and test developer: recognize and be aware of the effect of culture

Difference between test situation and the classroom

Cultural bias is not something that has to be avoided in the classroom

But: teachers should use cultural miscommunications to learn from it

Intercultural competence!

Teacher and test developer: recognize and be aware of the effect of culture

Tips: How to make students interculturally competent? Three important principles: Perspective Context Dialogue

Three important principles:

Perspective

Context

Dialogue

DISCUSSIE / VRAGEN What is your experience with cultural bias in your own classroom/test practice? Can you give an example? How do you manage?

What is your experience with cultural bias in your own classroom/test practice? Can you give an example? How do you manage?

LITERATUURVERWIJZINGEN Van Baalen, C., Beheydt, L. & Van Kalsbeek, A. (2003). Cultuur in taal. Interculturele vaardigheden voor docenten Nederlands aan anderstaligen . Utrecht: Nederlands centrum buitenlanders, 167p.

Van Baalen, C., Beheydt, L. & Van Kalsbeek, A. (2003). Cultuur in taal. Interculturele vaardigheden voor docenten Nederlands aan anderstaligen . Utrecht: Nederlands centrum buitenlanders, 167p.

DO INDONESIAN MAKE MOBILE CALLS AND DO RUSSIAN GO ON HOLIDAY? NHTV Breda – januar 2008 Katholieke Universiteit Leuven Universiteit van Amsterdam E.Teijsse-Jongkind @uva.nl [email_address] Language testing and culture The certificate of Dutch as a foreign language