The convenience and immediacy of translation apps is appealing, however this needs to be balanced against the need for quality translation. This study provides the amongst the first empirical evidence internationally on the extent and nature of the use of translation apps in health care settings. The findings may inform guideline development and policy responses.

1. It’s already happening: Use of translation
apps and websites in health care settings
Results of a study across five NSW Local Health Districts
Ben Harris-Roxas,1,2 Lisa Woodland,3,1 Joanne Corcoran,3 Jane Lloyd,1,4
Mark Harris,1 Rachael Kearns,1,2 Iqbal Hasan1
1. Centre for Primary Health Care and Equity, UNSW Sydney, Sydney
2. South Eastern Sydney Research Collaboration Hub (SEaRCH), Primary, Integrated
and Community Health, South Eastern Sydney Local Health District , Sydney
3. Priority Populations Unit, Primary, Integrated and Community Health, South Eastern
Sydney Local Health District, Sydney
4. Health Equity Research Development Unit, Sydney Local Health District, Sydney.

2. The issue
Use of translation apps and
websites in health settings
appears to be increasing.
Machine translation may be
inaccurate for health information,
and is markedly less accurate for
some languages.

3. TAB B

4. Recurrent neural networks
• Used by almost all translation apps and
websites
• A piece of software that functions as a
predictive model
• They need big training datasets

6. Image: Sabine Hossenfelder, Frankfurt Institute for Advanced Studies

7. Recurrent neural networks
• Neural networks typically have hundreds
or thousands of neurons
• The human brain has 100 billion (but
RNNs are faster)
• The hidden layers may be hard for us to
understand and interpret – and may
change over time
• They optimise for a defined output

9. Image: Sabine Hossenfelder, Frankfurt Institute for Advanced Studies

10. An example
“You need to drink some fluids
before your colopyopause that
will open your eyes and
remove from stool so that we
can see your face face.”

11. Potential uses
During clinical interactions
Written resources
General health information
Website information
It is unclear who’s initiating use and
in what circumstances

12. The context Most research on the development and
accuracy of machine translation (Birch et al 2016,
Koponen et al 2016, Patil and Davies 2014)
Some on types of apps (Panayiotou et al 2019)
Less on use within health care settings
(Anazawa et al 2013, Michael et al 2013)
2017 survey of 698 staff in three NSW
Local Health Districts/Specialty Networks
found 18% had used a translation app
Updated NSW Health Policy Directive in
December 2017 (NSW Health 2017)
Anazawa, R., et al. (2013). "Evaluation of online machine translation by nursing users." CIN: Computers, informatics, Nursing 31(8): 382-387.
Birch, A., et al. (2016). "HUME: Human UCCA-based evaluation of machine translation." arXiv preprint arXiv:1607.00030.
Koponen, M. (2016). "Is machine translation post-editing worth the effort? a survey of research into post-editing and effort." The Journal of Specialised
Translation 25: 131-148.
Michael, J., et al. (2013). "Development of a Translation Standard to support the improvement of health literacy and provide consistent high-quality information."
Australian Health Review 37(4): 547-551.
NSW Ministry of Health. Interpreters – Standard Procedures for Working with Health Care Interpreters [Policy Directive PD2017_044]. 2017.
Panayiotou, A., Gardner, A., Williams, S., Zucchi, E., Mascitti-Meuter, M., Goh, A.M.Y., You, E., Chong, T.W., Logiudice, D., Lin, X., Haralambous, B., Batchelor,
F., 2019. Language Translation Apps in Health Care Settings: Expert Opinion. JMIR mHealth and uHealth 7, e11316.
Patil, S. and P. Davies (2014). "Use of Google Translate in medical communication: evaluation of accuracy." BMJ 349: g7392.

13. Research question
What is the nature and extent of
translation app and website use
in state funded health care
services in NSW?
Staff survey and semi-structured
qualitative interviews

14. Survey sample 1,558 respondents
5 Local Health Districts
2.0% response rate
80.3% female, 18.2% male
70.7% clinical staff, 29.3% non-clinical
Interview sample 24 participants
5 Local Health Districts
Range of health professions, managers and
senior managers/policy roles

15. Survey findings
App users
33.6% had used a translation app or
website in a clinical encounter (n=516)
Of these:
75.4% had used a translation app within
the past 12 months (57.4% in past 3
months)
Clinicians initiated the use of a
translation app in two-thirds of their most
recent experience (66.8%)
72.8% has most recently used an app
between 8:30am and 5pm
42.6% used an app after a request for a
professional interpreter had been made.

16. Survey findings
App users
Younger
44.2% aged under 40 had used translation
apps, compared to 28.3% aged 40+.
χ(1)=40.181, p<0.001
Male
43.0% of males had used translation apps,
compared to 28.3% of people who identified
as female.
N.B. sample 80% female, χ(1)=10.676, p=0.001
Less experienced
39.6% who had worked for NSW Health for
less 10 years had used translation apps,
compared to 30.1% who had worked for
10+ years. χ(1)=14.759, p<0.001
Clinical staff
41.8% of clinical staff had used translation
apps, compared to 16.4% of non-clinical
staff. χ(1)=87.062, p<0.001

17. Survey findings
App users by profession
63.50%
41.9%
29.8%
17.8%
0%
10%
20%
30%
40%
50%
60%
70%
Medical Nursing/midwifery Allied health Administration
Proportion within profession using apps (%)
χ(10)=151.935, p<0.001

18. Survey findings
App users by setting
50.5%
45.0%
42.3%
39.3%
30.7%
11.9%
0%
10%
20%
30%
40%
50%
60%
Proportion of respodents within setting (%) χ(10)=89.588, p<0.001

19. Survey findings
Perceptions of app use
93.4% rated the translation app as very
useful or useful.
57.8% rated the risk of misinterpretation as
low or none.
33.5% rated the translation as accurate or
very accurate.

20. Interview findings
Risks being weighed up: Inaccuracy and time
“But the patient expressed himself in a way that I felt that I
almost certainly understood. It was congruent with what is
his hand gestures and his behaviour was I thought I
understood the response. But it was a very simple
communication. I think anything that was about symptoms
would be very difficult to, to rely on a translation app.”

21. Interview findings
Risks being weighed up: Inaccuracy and time
“We weren't able to book a face to face Thai interpreter on
time, and our patient was already on our bed so we weren't
able to physically pull the phone over to her dial the phone
interpreter and wait for them because there is also a chance
that we might not be able to get a phone Thai interpreter
either. Plus the use was only for some basic questions, so
we thought it would be okay.”

22. Interview findings
Risks being weighed up: Inaccuracy and time
“I think definitely “time-constraints” is a factor. We can't call
interpreter service on the phone every single day to ask a
patient every single day, and some patients just require a
little bit more communication than others, and we can't rely
on interpreters services every time when a patient has a
question, and they want it to be answered right there right
then, or if the patients really busy and they need to leave
very soon but they have questions in their mind. We can't
provide them with an interpreter every single time. Basically I
think it’s down to the best thing that we can do.”

23. Interview findings
Risks being weighed up: Inaccuracy and time
“Different cultures explain things differently. There's no direct
translation you have to understand a language, you can't
just use a word. So you can't type in a word and expect it,
guaranteed accurate.”

24. Interview findings
Risks being weighed up: Inaccuracy and time
“Anything that's emotive or involves more risk, I think, it's
imperative that you have accurate interpreting. But I think in
health generally it's really important to be accurate in your
language translations. I don't think non-human ways of doing
it are good enough at the present.”

25. Discussion Use appears to be widespread in state
funded health care services in NSW.
In most cases clinicians are initiating use.
Use may be growing. In the 2017 survey
18% reported app use. For the 301
respondents from a similar (but not exactly
the same) sample area in this survey,
38.2% reported app use.
Contradiction about perception that app
translation is not accurate, but that the
potential risks of this are low.
People are making trade-offs.
Limitations Low response rate.
Potential for self-censorship, actual rates of
use may be higher.

26. What remains unknown?
Is there variability in adoption in other
settings?
What are health consumers, family and
carers’ experiences and perceptions?
Is a more integrated research approach
may be required?
(Intersection of data science,
communication and linguistics, psychology,
and health and social service delivery)

27. The difficulty in developing
feasible and realistic policy
responses and guidance

28. Privacy and data governance

29. O’Leary, D. E. (2008). Gartner’s hype cycle and information system research issues. International Journal of Accounting Information Systems,
9(4), 240–252. https://doi.org/10.1016/j.accinf.2008.09.001
Where are we up to?

30. Where are we up to?
May, C. R., Cummings, A., Girling, M., Bracher, M., Mair, F. S., May, C. M., … Finch, T. (2018). Using Normalization Process Theory in
feasibility studies and process evaluations of complex healthcare interventions: A systematic review. Implementation Science, 13(1), 80

31. Deep learning
(or poor metaphors?)
“At the same time, deep learning has no good way to
incorporate background knowledge. A system can learn to
predict that the words wallet and safe place occur in
similar kinds of sentences ("He put his money in the
wallet," "He put his money in a safe place"), but it has no
way to relate that to the fact that people like to protect
their possessions.”
Source: Marcus G, Davis E. (2019) Rebooting AI: Building Artificial Intelligence We Can Trust,
Pantheon: New York.
Image: Marketa

32. “Super-linguists” will be needed
to trim the linguistic engines
and, eventually, override the
recommendations of machines.
For a long time still, complex
semantics and the interpretation
of text passages will remain the
art of human involvement in
processing the input and output.
Technology is not set to become
autonomous in the medium
term. It can serve people very
well if applied with boldness and
creativity, as well as with
responsibility.”
Source: Svoboda, T. (2017). No linguistic borders
ahead? Looking beyond the knocked-down
language barrier. TranscUlturAl: A Journal of
Translation and Cultural Studies, 9(2), 86–108.
https://doi.org/10.21992/T93Q0FImage: Robert Couse-Baker
The future?

33. Moving beyond the
transmission model
Health literacy
The importance of community and
culture
Genuine person-centred
approaches
Image: Olaf Eichler

34. What’s at stake?

35. Acknowledgements
Funders
Priority Populations Unit, South Eastern Sydney Local
Health District; NSW Ministry of Health
Site investigators
Sue Buckman, Vesna Dragoje, Eva Melham, Bradley
Warner, Ashley Young
Collaborators
SESLHD; Health and Social Policy Branch, NSW
Ministry of Health; WSLHD Translation Unit; SLHD,
SHCIS; SWSLHD; Refugee Health Service, HNELHD,
Multicultural Health Service, NSW Multicultural Health
Communications Service

36. Thank you
Ben Harris-Roxas
Email b.harris-roxas@unsw.edu.au
These slides are available at
https://www.slideshare.net/benharrisroxas