This document summarizes Tim McNamara's presentation on occupation-specific language assessments. It discusses two studies conducted as part of a larger research project on developing language proficiency standards for non-native English speaking health professionals. The first study identified criteria used by health professionals to evaluate clinical communication skills through analyzing feedback on interactions between trainees and patients. It found that criteria valued by health professionals differ from traditional linguistic criteria assessed in language tests. The second study investigated applying these newly identified, interaction-focused criteria to the speaking component of the Occupational English Test (OET). It found assessors could reliably apply the new criteria and they capture a different but related dimension of competence compared to existing criteria. Implementation would change pass

1. Plenary 2
Tim McNamara
School of Languages and Linguistics, University of Melbourne

2. What language assessment
theorists and practitioners have
learned about Occupation-
specific Language Assessments
Tim McNamara
School of Languages and Linguistics &
Language Testing Research Centre
University of Melbourne

3. Working with language across boundaries
What is the role of language in the competence
of health professionals?
Do language professionals and health
professionals (‘domain experts’) share an
understanding of this role?
How can language professionals come to
understand the thinking of domain experts?

4. How can language specialists understand the
the specific purpose domain?
The problem of reconciling the different
perspectives of language and non-
language professionals is a perennial
concern for Languages for Specific
Purposes practitioners.
Basturkmen & Elder (2004, p. 677)

5. Occupational English Test (OET)
Specific-purpose English language and
communication (ESP) test for migrant health
professionals in 12 professions whose training has
not been in English
4 skills test designed to assess the readiness of candidates
to manage the communicative tasks of the health workplace
– Speaking, writing – communicative tasks specific to each
profession
– Listening, reading – communicative tasks common to all health
professions
– Recognized for professional registration of migrant health
professionals in Australia, New Zealand and Singapore

6. The policy question…
Need to ‘get it right’ about who to admit, who to
exclude
Involves balancing
patient safety standards vs
not creating unreasonable restrictions on IMGs’
access to clinical experience
Evidence that the decisions are not always right –
aggrieved failing candidates
complaints from clinical supervisors
By what criteria should candidates be judged?
2 studies: on speaking criteria, on writing criteria

7. Australian Research Council Linkage grant (LP0991153)
Ethics approval for study given by University of Melbourne
Languages & Linguistics Human Ethics Advisory Group
7
Towards improved healthcare communication:
Development and validation of language
proficiency standards for non-native English
speaking health professionals
Catherine Elder, Robyn Woodward-Kron, Tim McNamara,
Elizabeth Manias, Geoff McColl, Gillian Webb
John Pill, Sally O’Hagan, Diana van Die

8. OET Speaking sub-test
Test task: role-plays between health professional and patient
– profession-specific
Test content: common situations encountered in the workplace
– accuracy and currency confirmed by health professional advisors
Assessment: linguistic criteria applied by language professionals to
audio recorded performances of interaction
Overall Communicative Effectiveness
Intelligibility
Fluency
Appropriateness of Language
Resources of Grammar and
Expression

9. SAMPLE MATERIALS - SPEAKING SUB-TEST
(PHYSIOTHERAPISTS): Candidate’s card
SETTING: Hospital clinic
PATIENT: An elderly person who is recovering from a stroke
(CVA). The patient is making slow progress in learning to walk
again.
TASK: Talk to the patient about the following pieces of
equipment:
• a wheelchair
• a walking frame
• a walking stick.
– Explain the advantages and disadvantages of each one.
– You would like the patient to be as independent in his/her
movements as possible. You feel the frame is not
appropriate.
– You want the patient to have a stick. You do not want the
patient to have a wheelchair at this stage.

10. SAMPLE MATERIALS - SPEAKING SUB-TEST
(PHYSIOTHERAPISTS): Role-player’s card
SETTING: Hospital clinic
PATIENT: You are an elderly person who is recovering from a
stroke. You feel you are making painfully slow progress, and
don't really expect to be able to walk again.
You feel you should be allowed to have a wheelchair.
TASK: Ask the physiotherapist when you will be given a
wheelchair.
Insist on your need for this equipment. Explain that you feel that
the painful exercises you are doing at the moment are pointless,
and that you are pessimistic about your chances of making real
progress.
Be difficult!

11. 3 yr study of ‘indigenous’ assessment criteria for
judging communication in medical workplace
Investigation of feedback by clinical educators on
clinical spoken interaction between trainee health
professionals (HPs) and patients:
to investigate what qualities are valued by health
professionals in spoken clinical communication
and
to see if these should be and can be included in
a test of English for spoken clinical
communication
Towards improved healthcare communication
(Australian Research Council grant, 2009-2013)

12. Jacoby (1998): Indigenous assessment
Indigenous assessment – naturally
occurring, routine instances where
communication is judged
Indigenous assessment criteria – criteria
which emerge in these contexts

13. Investigating views of content experts
Jacoby 1998:
Studied feedback from expert physicist to junior
physicists rehearsing conference papers
Presenters both native and non-native speakers
of English S and NNS (some with limited
language proficiency)
No distinction between these groups in
feedback – occasional remarks on language
Organization, timing etc more important

14. Towards improved healthcare communication
(Australian Research Council grant, 2009-2013)
Phase 1
Identifying criteria indigenous to workplace context
Translating them into assessment criteria
Phase 2
Implementing new criteria
Investigating how they function
Phase 3
Setting cut-scores (e.g., pass/fail)
3 Professions: Medicine, Nursing and Physiotherapy
WHAT
HOW
HOW
MUCH

15. Phase 1
Phase 1
Identifying criteria indigenous to the workplace context
for judging the adequacy of clinical communication skills
Eliciting and analysing health professionals HP
judgements of actual clinical communication
What do (HPs) themselves value in communication with
patients?
Result: model of what HPs value (Pill 2013) – shows
strong differences from linguistic criteria
Translating this into assessment criteria to be used in
judging performances on the Occupational English Test

16. Model of what health professionals value in
patient communication (Pill 2012)

17. OET Speaking assessment criteria
Current
language-related
criteria
Proposed
interaction-related
criteria
Intelligibility
Clinician
Engagement
Fluency
Appropriateness of
Language
Management of
InteractionResources of
Grammar and
Expression

18. Proposed new criteria
Clinician Engagement
This criterion refers to the ability to use language to demonstrate a professional
manner towards and awareness of the patient.
You should consider whether the candidate demonstrates:
• a suitably positive manner towards the patient (i.e., attentive, tolerant, respectful,
non-judgemental)
• awareness of the patient’s situation, narrative, opinions and expectations,
acknowledging the patient’s concerns, needs and emotions.
(4)
• Professional manner generally strong and positive.
• Generally high level of awareness of the patient.
(3)
• Professional manner sufficiently positive.
• Generally adequate level of awareness of the patient.
(etc.) …
18
Clinician Engagement
Management of Interaction

19. Phase 2: Can this model inform OET criteria?
What is at stake in the statistical analysisCan assessment of these two new
aspects be handled successfully by
language professionals?
How do the new criteria relate to the
existing criteria?
Are the new criteria measuring a new
aspect of the same construct or
introducing a different construct?

20. New criteria represent related but somewhat different
dimension of competence
The addition of two new criteria in a revised test would
represent more of what health professionals value in
clinical spoken communication with patients/relatives
Existing OET (language-focused) assessors can be
trained to apply these new criteria consistently (though
further training necessary)
Two options to report the speaking sub-test results:
• As two separate grades for the new and existing criteria;
• As a single grade for existing criteria supplemented by a
diagnostic report relating to the scores given on the new
criteria.
Summary of findings – Phases 1 & 2

21. Findings of Phase 3
Fewer test takers would pass using new criteria
But more of those would achieve an A (“strong”) grade
Cut-scores for nursing were a bit lower than for medicine
Moderate to good correlation between
– Health professionals’ (HPs’) holistic judgement
– OET assessors’ grading using six criteria
(2 interactional and 4 language-related)
The new criteria appear to capture what matters to HPs

22. Phase 2:
The two additional criteria allow claims that a revised
test would represent more of what health professionals
value
OET (language-focused) assessors can be trained to
apply these criteria consistently
Phase 3:
Implementation of new criteria would change current
outcomes for some candidates
Promises better decisions about who to admit, who not
to admit, to clinical practice
Summary and conclusion

23. Australian Research Council Linkage grant (LP130100171)
Ethics approval for study given by University of Melbourne
Languages & Linguistics Human Ethics Advisory Group
23
Toward improved quality of written patient
records: Language proficiency standards for
non-native English speaking health
professionals
Tim McNamara, Ute Knoch, Robyn Woodward-Kron, Catherine
Elder, Elizabeth Manias, Eleanor Flynn, Barbara Zhang
Sharon Yahalom, Simon Davidson, Annemiek Huisman

24. OET Writing sub-test
Test task: Letter of referral/discharge
Test content: Based on case notes
Assessment: Linguistic criteria applied by language professionals
Overall Task Fulfilment
Appropriateness of Language
Comprehension of Stimulus
Linguistic Features (Grammar & Cohesion)
Presentation Features (Spelling, Punctuation & Layout)

25. Overview of the project
Phase 1
• Interviews with stakeholders who
contribute to or read medical records
Phase 2
• Establishing healthcare professional
relevant criteria
Phase 3
• Applying the professionally relevant
criteria to the OET writing task
Phase 4
• Setting minimum standards on the OET
writing task

26. Phase 2
Conducted workshops in which doctors were asked to
comment on the strengths and weaknesses of a range of
handover documents
Participants: 61 health professionals (doctors, nurses,
health information managers)
18 experienced doctors from a range of sub-disciplines and
contexts
Average years of experience: 25 years
Average years as educators: 17 years
Stimulus material: 10 handover documents (referral letters
and discharge summaries) selected from 200 patient
medical records

27. Referral letter

28. Discharge summary

30. Sample medical record

31. Extraction of all referral
letters and discharge
summaries
Preselection of
documents
Researcher meeting
for final selection
Methodology

32. Data collection procedures
Workshops conducted in small groups or individually
Audio recorded
Data analysis
Audio recordings transcribed
Coded for salient themes and sub-themes
Methodology

33. Results

34. 1. Appropriateness
Writer Text Reader

35. 1. Appropriateness
Appropriateness of the writing to the text
type/ style of the document
Any clinician would be able to understand it but it
is not the sort of language you want in a
discharge summary (DW1, M2, D1,2,3)
It is almost like somebody has had a private
investigator following him but it doesn’t scream
of medical assessment
(DW1, M22, D4,5)

36. 1. Appropriateness
Appropriateness of writing to audience
I mean give it a medical term because it is going to the
GP (DW1, M12, D1,2,3)
P: I also find it quite difficult sometimes when you are
referring to a consultant, I don’t like to be saying ‘Well
this is what I would like’ because sometimes
consultants don’t like being told what to do
P: Yeah, don’t tell me what to do, I will decide what to
do, so therefore often I will say, ‘I would be pleased
for your expert management’ or something like that.
(DW2, R13, D11,12)

37. 1. Appropriateness
Appropriateness of the writing to the
sub-discipline of the writer
… when I went to psych that is what I was doing, I
was doing bullet points and they said, “No, you have
to write novels”. So then I started writing novels and
then coming back here, I am like I don’t have to write
novels anymore. (DW2, M25, D8,9,10)

38. 2. Conciseness
• Conciseness:
So it is succinct which is also highly valued and as you get
more experienced you get more succinct.
(DW1, R66, D4,5)

39. 2. Conciseness
This discharge summary falls into the trap of trying to be
succinct with bullet points but at the same time being very
verbose (DW2, M12, D8,9,10)

40. 3. Sufficiency
Sufficiency:
You have got the information mostly that you need
here and so he/she has done that well (DW2, M20,
D11, 12)
P: Um, is there too much information sometimes?
P: No, I think as an initial letter because this is the first
time they have seen him in clinic, I think that is perfect.
P: and this would give me all the information I need
about his background (DW4, M25, D14)

41. 4. Accuracy of content
P: when you get particular
things like antibiotics that
they would have prescribed
five years ago and it is still
listed on their medication
list
P: so I would find that a
little bit embarrassing to
send it. It reflects poorly on
you if you send off a letter
which is incorrect.

42. 5. Clarity
Clarity of content
– Being explicit
– Making purpose of letter clear (in case of referrals)
– Making request or follow up plans clear (both referrals
and discharge summaries)
– Prioritizing information
It is not clear who is doing what or even what has
really happened. It is just a jumble of thoughts
really (DW1, R66, D1,2,3)
Towards the end I think when you really want to lay
it out, clarity is really crucial (DW1, M25, D1,2,3)

43. 5. Clarity
Clarity of organisation & presentation
– Logical organisation of content
– Prioritization of information
– Use of paragraphing, dot points, spacing when
appropriate
– Legibility & general ‘tidiness’
The information is there and the information is good
but actually that still doesn’t do the job because of
how it is presented (DW2, R13, D8,9,10)
So I think it is very well structured in terms of you
could pick this up and get a good picture of the
patient and what they look like (DW4, M22, D14)

44. 5. Clarity
Clarity of language
– Grammatical accuracy, spelling, sentence
structure were all mentioned
So the grammar and spelling and English is not 100%
but that is not an issue in terms of understanding what is
there (DW4, R36, D14)
You could ignore the sentence structure, provided what
was written down was comprehensive and useful (DW2,
R36, D8, 9, 10)

45. 5. Clarity
Clarity of language
– Use of acronyms
– Use of technical language
P: I don’t like acronyms, I don’t know what BRAF is?
P: no
P: there is a reference to BRAF, which is obviously an abbreviation
but within the oncology field (DW2_M25_D11, D12)
P: But it doesn’t read very easily because there is a lot of acronyms,
there is a lot of shortcuts (DW2_R13_D8, 9, 10)

46. Discussion – what doctors value in handover
documentation
Appropriateness
Writer
Text
Reader

47. Discussion – what doctors value in handover
documentation
Conciseness
&
Sufficiency
Appropriateness
All key information
present
No irrelevant
information

48. Discussion – what doctors value in handover
documentation
Clarity
Conciseness
&
Sufficiency
Appropriateness
content
organisation
presentation
language

49. Discussion – what doctors value in handover
documentation
Clarity
Conciseness
&
Sufficiency
Appropriateness
Accuracyofcontent

50. Discussion
Values of domain experts cannot always be directly
translated  do not all transfer into the (more limited)
language testing domain
Many of the aspects mentioned by the domain experts are
represented in the current criteria
Some aspects not present
Grouping might have to be re-examined
Weighting will have to be re-examined
Next steps:
Examining how well descriptors are reflected in our empirical data
Develop checklist indicators
Apply checklist indicators to OET writing data

51. Strong and weak performance tests
Two types of test:
‘in the strong sense, tasks will represent real-world tasks, and
performance will primarily be judged on real-world criteria, that
is, the fulfilment of the task set. Aspects of language ability as
such may or may not be assessed at all’
‘in the weak sense, the focus is on language performance.
Task may resemble or simulate real-world tasks … however,
the capacity to perform the task is not actually the focus of
assessment. Rather, the purpose … is to elicit a language
sample so that second language proficiency … may be
assessed.’
McNamara, T. F. (1996). Measuring second language performance. Harlow: Longman. (pp. 43-44)

52. [The body] responsible for the OET is required by law to
assess the English language proficiency of overseas-trained
health professionals, but is not permitted to make
assessments of their professional competence (McNamara,
1996, p. 40)
(Impossible) separation of communicative skill and
professional competence
Assessment of communicative skill in hands of language
professionals (language teachers, language testing experts)
But are language assessors and clinicians oriented to the
same qualities in communication in interactions between
clinicians and patients?
Construct issue: Divided responsibility for
assessment

53. © Copyright The University of Melbourne 2011

54. Plenary 2
Tim McNamara
School of Languages and Linguistics, University of Melbourne