"I understand your concern for the environment. However, banning all plastics may have unintended consequences that harm people's livelihoods and well-being. Perhaps we could find an approach that reduces plastic pollution while supporting people's needs." Patient: blah blah blah...

1. Talking with virtual humans
Willem-Paul Brinkman
Interac(ve Intelligence Group
Del3 University of Technology

2. Mission: Developing and understanding empirically grounded
computer models that generate effec(ve and acceptable computer
behaviour for behaviour change

3. Computer support for anxiety
disorder treatment at home (CATCH)

4. 1.
Human controlled versus automa0c
dialogue

5. Social phobia – speech recognition

6. Public Speaking
ter Heijden, N., and Brinkman, W.-P. (2011). Design and evalua(on of a virtual reality
exposure therapy system with automa(c free speech interac(on. Journal of CyberTherapy
and Rehabilita0on, 4(1), 41-55.

7. 1
2
3
4
5
6
7
Speech detector Limited speech
recogni(on
Speech
recogni(on
Human control
Ccomputer vs human control

8. 2.
Priming and dialogues with a virtual
human

9. Qu, C., Brinkman, W. P., Wiggers, P., & Heynderickx, I. (2013). The effect of priming pictures and
videos on a ques(on-answer dialog scenario in a virtual environment. Presence: Teleoperators
and Virtual Environments, 22(2), 91-109.

10. Human – Human Dialogue
0
1
2
3
4
Unrelated Related
Keywords
Pictures
Unrelated video Related video

12. Human – Virtual Human dialogue
0
1
2
3
4
Unrelated Related
Keywords
Pictures
Unrelated video Related video

13. 3.
Listening versus speaking

14. Qu, C., Brinkman, W. P., Ling, Y., Wiggers, P., & Heynderickx, I. (2013). Human
percep(on of a conversa(onal virtual human: an empirical study on the effect of
emo(on and culture. Virtual Reality, 1-15.

16. Speaking and listening

18. Qu C, Brinkman W, Ling Y, Wiggers P, Heynderickx I, (2014). Conversa(ons with a virtual
human: Synthe(c emo(ons and human responses, Computers in Human Behavior, 34

20. 4.
Virtual Reality Exposure therapy

21. Virtual Reality Exposure Therapy for Social Anxiety Disorder:
A Randomized Controlled Trial
Page L. Anderson
Georgia State University
Matthew Price
University of Vermont
Shannan M. Edwards, Mayowa A. Obasaju, Stefan K. Schmertz, Elana Zimand, and Martha R. Calamaras
Georgia State University
Objective: This is the first randomized trial comparing virtual reality exposure therapy to in vivo
exposure for social anxiety disorder. Method: Participants with a principal diagnosis of social anxiety
disorder who identified public speaking as their primary fear (N ϭ 97) were recruited from the
community, resulting in an ethnically diverse sample (M age ϭ 39 years) of mostly women (62%).
Participants were randomly assigned to and completed 8 sessions of manualized virtual reality exposure
therapy, exposure group therapy, or wait list. Standardized self-report measures were collected at
pretreatment, posttreatment, and 12-month follow-up, and process measures were collected during
treatment. A standardized speech task was delivered at pre- and posttreatment, and diagnostic status was
reassessed at 3-month follow-up. Results: Analysis of covariance showed that, relative to wait list, people
completing either active treatment significantly improved on all but one measure (length of speech for
exposure group therapy and self-reported fear of negative evaluation for virtual reality exposure therapy).
At 12-month follow-up, people showed significant improvement from pretreatment on all measures.
There were no differences between the active treatments on any process or outcome measure at any time,
nor differences on achieving partial or full remission. Conclusion: Virtual reality exposure therapy is
effective for treating social fears, and improvement is maintained for 1 year. Virtual reality exposure
therapy is equally effective as exposure group therapy; further research with a larger sample is needed,
however, to better control and statistically test differences between the treatments.
Keywords: virtual reality exposure therapy, social anxiety disorder, social phobia, cognitive behavioral
therapy
Virtual reality exposure therapy (VRE) for the treatment of
anxiety disorders has received considerable attention (Parsons &
Rizzo, 2008). During VRE, a person encounters a feared stimulus
in a computer-generated environment, often through the use of a
head-mounted display and motion tracker that allows for multi-
sensory input and natural movement. Several advantages of VRE
have been noted—both clinical, such as treatment acceptability
(Emmelkamp, 2005), and methodological, such as the ability to
conduct exposure in a tightly controlled environment (Shiban,
however, highlights that empirical support for VRE varies across
anxiety disorders (Meyerbröker & Emmelkamp, 2010). Specifi-
cally, randomized controlled trials support the efficacy of VRE
with fear of flying (e.g., Rothbaum, Hodges, Smith, Lee, & Price,
2000) and acrophobia (e.g., Emmelkamp et al., 2002) and show
that it is equally effective as in vivo exposure. These phobias lend
themselves well to VRE; the feared stimulus is circumscribed and
contains powerful physical cues that can be produced within a
virtual environment. There is less controlled research on the use of
ocumentiscopyrightedbytheAmericanPsychologicalAssociationoroneofitsalliedpublishers.
leisintendedsolelyforthepersonaluseoftheindividualuserandisnottobedisseminatedbroadly.
Journal of Consulting and Clinical Psychology © 2013 American Psychological Association
2013, Vol. 81, No. 5, 000 0022-006X/13/$12.00 DOI: 10.1037/a0033559

22. Exposure to virtual social interactions in the treatment of social
anxiety disorder: A randomized controlled trial
Isabel L. Kampmann a, *
, Paul M.G. Emmelkamp c, d
, Dwi Hartanto b
,
Willem-Paul Brinkman b
, Bonne J.H. Zijlstra e
, Nexhmedin Morina a
a
Department of Clinical Psychology, University of Amsterdam, Weesperplein 4, 1018 XA, Amsterdam, The Netherlands
b
Interactive Intelligence Group, Delft University of Technology, Mekelweg 4, 2628 CD, Delft, The Netherlands
c
Netherlands Institute for Advanced Study, Meijboomlaan 1, 2242 PR, Wassenaar, The Netherlands
d
King Abdulaziz University, Abdullah Sulayman, 22254, Jeddah, Saudi Arabia
e
Department of Educational Sciences, University of Amsterdam, Nieuwe Achtergracht 127, 1018 WS, Amsterdam, The Netherlands
a r t i c l e i n f o
Article history:
Received 4 September 2015
Received in revised form
19 December 2015
Accepted 23 December 2015
Available online 29 December 2015
Keywords:
Virtual reality
Exposure therapy
Social anxiety disorder
Social phobia
Social interaction
a b s t r a c t
This randomized controlled trial investigated the efficacy of a stand-alone virtual reality exposure
intervention comprising verbal interaction with virtual humans to target heterogeneous social fears in
participants with social anxiety disorder. Sixty participants (Mage ¼ 36.9 years; 63.3% women) diagnosed
with social anxiety disorder were randomly assigned to individual virtual reality exposure therapy
(VRET), individual in vivo exposure therapy (iVET), or waiting-list. Multilevel regression analyses
revealed that both treatment groups improved from pre-to postassessment on social anxiety symptoms,
speech duration, perceived stress, and avoidant personality disorder related beliefs when compared to
the waiting-list. Participants receiving iVET, but not VRET, improved on fear of negative evaluation,
speech performance, general anxiety, depression, and quality of life relative to those on waiting-list. The
iVET condition was further superior to the VRET condition regarding decreases in social anxiety symp-
toms at post- and follow-up assessments, and avoidant personality disorder related beliefs at follow-up.
At follow-up, all improvements were significant for iVET. For VRET, only the effect for perceived stress
was significant. VRET containing extensive verbal interaction without any cognitive components can
effectively reduce complaints of generalized social anxiety disorder. Future technological and psycho-
logical improvements of virtual social interactions might further enhance the efficacy of VRET for social
anxiety disorder.
© 2015 Elsevier Ltd. All rights reserved.
Social anxiety disorder (SAD) is defined as the fear of one or
more social situations in which one might behave embarrassingly
and be negatively evaluated by others (DSM-V; American
therapy (CBT). CBT aims at modifying maladaptive cognitions and
behavior using both cognitive (e.g., cognitive restructuring) and
behavioural (e.g., exposure) strategies (Hofmann & Smits, 2008;
Contents lists available at ScienceDirect
Behaviour Research and Therapy
journal homepage: www.elsevier.com/locate/brat
Behaviour Research and Therapy 77 (2016) 147e156

23. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
• 19 scenario’s
• Each las(ng around 18 minutes
• Free speech
• Automa(c response virtual human

26. Kampmann, I. L., Emmelkamp, P. M., Hartanto, D., Brinkman, W. P., Zijlstra,
B. J., & Morina, N. (2016). Exposure to virtual social interac(ons in the
treatment of social anxiety disorder: A randomized controlled trial.
Behaviour Research and Therapy, 77, 147-156.

27. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
Dialogue Unit:
1. [avatar’s ques(on]
2. [pa(ent’s answer]
3. [avatar’s response]
• Each dialogue on average 78 dialogue
units
• Speech detec(on
• Keyword detec(on

28. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
8 general strategies to responses
without keyword detec(on
Example 1: generic response that fits to
any answer
[shop assistance] “Can you also specify
to me the price range that you’re aiming
for?”
[pa(ent] blah blah blah….
[shop assistance] “Well, that’s fine”.

29. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
Example 2: State an opinion based on
your beliefs, emo(on or percep(on
[avatar] “What makes you a good team
leader?”
[pa(ent] blah blah blah….
[avatar] “Yeah, but honestly I feel that
you’re not ready yet to become a good
team leader by judging your answer and
your current experience”.

30. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
Other strategies
• Use topics blocks in the story line to
avoid the repe((on of Q&A
• (dis)agree on what the pa(ent said
• State and aptude or emo(on towards
the answer
• Reflect on your original ques(on
• State(mis)understaning of pa(ent
answer and extend response with own
informa(on
• State an opinion

31. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
Dialogue Unit:
1. [Posi>ve | Nega>ve avatar’s
ques(on]
2. [pa(ent’s answer]
3. [Posi>ve | Nega>ve avatar’s
response]
Example Posi>ve Unit
[avatar] “I like to know your taste music,
what kind of music do you like?”
[pa(ent] blah blah blah…..
[avatar] “Cool! Nice taste of music!”

32. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
Dialogue Unit:
1. [Posi>ve | Nega>ve avatar’s ques(on]
2. [pa(ent’s answer]
3. [Posi>ve | Nega>ve avatar’s response]
Example Nega>ve Unit
[avatar] “I don’t think that you have a good
taste of music, but in case I’m wrong, can
you tell me what type of music you like?”
[pa(ent] blah blah blah…..
[avatar] “Mmm… as I have expected, you
know nothing about good music!”

34. Controlling Social Stress in Virtual Reality Environments
Dwi Hartanto1
*, Isabel L. Kampmann2
, Nexhmedin Morina2
, Paul G. M. Emmelkamp2,3
,
Mark A. Neerincx1,4
, Willem-Paul Brinkman1
1 Department of Intelligent Systems, Delft University of Technology, Delft, The Netherlands, 2 Department of Clinical Psychology, University of Amsterdam, Amsterdam,
The Netherlands, 3 Department of Psychology, King Abdulaziz University, Jeddah, Saudi Arabia, 4 TNO Human Factors, Soesterberg, The Netherlands
Abstract
Virtual reality exposure therapy has been proposed as a viable alternative in the treatment of anxiety disorders, including
social anxiety disorder. Therapists could benefit from extensive control of anxiety eliciting stimuli during virtual exposure.
Two stimuli controls are studied in this study: the social dialogue situation, and the dialogue feedback responses (negative
or positive) between a human and a virtual character. In the first study, 16 participants were exposed in three virtual reality
scenarios: a neutral virtual world, blind date scenario, and job interview scenario. Results showed a significant difference
between the three virtual scenarios in the level of self-reported anxiety and heart rate. In the second study, 24 participants
were exposed to a job interview scenario in a virtual environment where the ratio between negative and positive dialogue
feedback responses of a virtual character was systematically varied on-the-fly. Results yielded that within a dialogue the
more positive dialogue feedback resulted in less self-reported anxiety, lower heart rate, and longer answers, while more
negative dialogue feedback of the virtual character resulted in the opposite. The correlations between on the one hand the
dialogue stressor ratio and on the other hand the means of SUD score, heart rate and audio length in the eight dialogue
conditions showed a strong relationship: r(6) = 0.91, p = 0.002; r(6) = 0.76, p = 0.028 and r(6) = 20.94, p = 0.001 respectively.
Furthermore, more anticipatory anxiety reported before exposure was found to coincide with more self-reported anxiety,
and shorter answers during the virtual exposure. These results demonstrate that social dialogues in a virtual environment
can be effectively manipulated for therapeutic purposes.
Citation: Hartanto D, Kampmann IL, Morina N, Emmelkamp PGM, Neerincx MA, et al. (2014) Controlling Social Stress in Virtual Reality Environments. PLoS
ONE 9(3): e92804. doi:10.1371/journal.pone.0092804
Editor: Mel Slater, ICREA-University of Barcelona, Spain
Received November 26, 2013; Accepted February 25, 2014; Published March 26, 2014
Copyright: ß 2014 Hartanto et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits
unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This research is supported by the Netherlands Organization for Scientific Research (NWO), grant number 655.010.207. The funders had no role in study
design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing Interests: The authors have declared that no competing interest exist.
* E-mail: d.hartanto@tudelft.nl
Introduction
Social anxiety disorder, also commonly referred as social
phobia, is one of the most prevalent mental disorders [1]. People
with social phobia experience a strong fear of being judged
negatively by others and of being embarrassed in social situations
(e.g., talking to other people or eating or drinking in front of other
people). The gold standard to treat patients with social phobia is
cognitive behaviour therapy with the central component being
transport, or restaurants. At the start of the treatment, an anxiety
hierarchy of anxiety-arousing social situations is established. This
hierarchy is then used to order the VR situations the patient will
be gradually exposed to, starting with less anxiety-arousing
situations and eventually moving to more anxiety-arousing
situations as treatment progresses.
Even though several studies [3,4,10,11] have reported promis-
ing initial efficacy findings for VRET for social phobia, the used
VR systems mainly allow the therapist to control social anxiety
80
85
90
95
0% 25% 50% 75% 100%
Heart rate
Nega1ve ques1ons

35. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
Eye Gaze Body Posture
Kang, N., Brinkman, W. P., van Riemsdijk, M. B., & Neerincx, M. (2016). The design of
virtual audiences: No(ceable and recognizable behavioral styles. Computers in
Human Behavior, 55, 680-694.

36. • Virtual social
scenarios
• Dialogue
techniques
• Phobic stressors
• Anxiety feedback
loop
Virtual reality system
Virtual world
Behavioral stressors
Dialogue stressors
Anxiety Monitoring
Subjec(ve
Heart rate
Anxiety level
controller
Anxiety
Devia(on
+/-
Required anxiety level
Anxiety measured
Single anxiety
measurement
therapist
pa(ent

37. 5 10 15 20
60708090100110
Exposure number across sessions
Heartrate
Heart rate even 4 minutes
• 204 Heart rate samples
• Sign main effects: sessions (between sessions 2-9), and
exposure order number (within a session 1-3)
Pilot Study
Hartanto, D., Brinkman, W. P., Kampmann, I. L., Morina, N., Emmelkamp, P. G., & Neerincx, M. A.
(2015). Home-Based Virtual Reality Exposure Therapy with Virtual Health Agent Support. In Pervasive
Compu0ng Paradigms for Mental Health (pp. 85-98). Springer Interna(onal Publishing.

38. Thanks for your awen(on
ICT is successful when it supports your need
for change, and fails when you need to
change for its support