CBT skills usage after receiving internet-delivered cognitive behavioral therapy (iCBT) was assessed to understand its role in maintaining treatment effects. 77 participants completed measures of depression, anxiety and functioning before, immediately after, and 3 months after iCBT. Symptoms significantly improved post-treatment and were maintained at follow-up. At follow-up, participants reported frequently using cognitive and behavioral CBT skills. While CBT skills usage predicted immediate outcomes, it did not predict maintenance of effects at 3-month follow-up, suggesting other factors may be involved in long-term outcomes.

1. Internet-Delivered Cognitive-
Behaviour Therapy for Anxiety and
Depression:
CBT Skills Usage and its Role in
Maintaining Outcomes
By Nora Eilert
Supervised by
Dr Ladislav Timulak
Dr Derek Richards

2. Outline
 Introduction
 Aims & Hypothesis
 Methodology
 Results
 Discussion
 Limitations
 Conclusion & Future Research

3. Introduction
 Internet-delivered Cognitive Behaviour Therapy (iCBT)
has been found to be effective in treating various
psychological disorders (i.e. Hedman et al., 2013; Richards, Richardson,
Timulak & McElvaney, 2015; Tulbure et al., 2015; Wagner, Horn, & Maercker, 2014)
 Gaps in the iCBT literature remain around long-term
effectiveness of iCBT & mechansims of change involved
(Andersson, 2016; Mogoașe, Cobeanu, David, Giosan & Szentagotai, 2016)

4. Introduction
 iCBT follow-up outcome research is insufficient & little is
know about which factors promote lasting change
 Understanding mechanisms of change involved in iCBT
will be essential in ruling out a lack of reliable change as
cause of inconsistent long-term effect findings
(Andersson, 2016; Mogoașe, Cobeanu,
David, Giosan & Szentagotai, 2016)

5. Introduction
 Little research available into mediators of iCBT and CBT
follow-up outcomes
 Currently most supported mediator of iCBT and CBT
follow-up outcomes is CBT skills usage
 CBT skills usage refers to how much individuals use the
techniques and strategies they learned during CBT
(i.e. French et al., 2016; Halmetoja,
Malmquist, Carlbring & Andersson, 2014)

6. Aims
 To further support the long-term effectiveness of iCBT for
anxiety and depression by exploring how effects are
maintained into follow-up
 To assess CBT skills usages after receiving iCBT in general
and its role in maintenance of outcomes in particular

7. Research Questions &
Hypothesis
 What does CBT skills usage after iCBT look like?
 How often do participants use CBT skills?
 What are participants experiences around using CBT skills?
 Does CBT skills usage predict maintenance of effects
after iCBT?
 In line with research on immediate iCBT outcomes (Forand et al.,
2017; Terides et al., 2017), CBT skills usage was hypothesised to predict
follow-up outcomes of iCBT as well

8. Method
 The current study was nested in large Randomised
Controlled Trial (RCT) conducted by SilverCloud Health
and the Berkshire NHS Trust in the UK
 Berkshire NHS Trust operates a stepped care model with
iCBT at step two
 SilverCloud Health is a provider of online therapeutic
solutions widely used across UK primary mental health
services

9. The parent study
 Single-blinded, parallel-groups RCT to test effectiveness
and cost-effectiveness of iCBT for anxiety & depression
against a wait-list control
 Randomisation 2:1
 Includes follow-up assessments of the experimental
group at 3-, 6-, 9- and 12-months from baseline
 Trial registration on Clinicaltrials.gov under identifier
NCT03188575
(Richards et al., 2018)

10. The current study
 Within-group design, measuring psychological symptoms
and functioning before and after 8 weeks of iCBT and at
3-month follow-up in the experimental group only
 CBT skills usage measured through mixed methods at
3-month follow-up

11. Participants
 All adult Berkshire NHS Trust service users at Step 2 were
eligible to participate
 Inclusion & exclusion criteria
 Anxiety and/or depression (as per Patient Health Questionnaire-
9 & Generalised Anxiety Disorder-7)
 18+ & suitable for iCBT
 No suicidal ideation, psychotic illness, current treatment,
alcohol/drug misuse or organic mental health disorder

12. Participants
Sample for the current study = 79 participants
Completed 3-month follow-up measures
79 by the end of the February 2018
Assigned 3-month follow-up measures
96 by mid-February 2018
Randomised to experimental group
197 by the end of February
Consented to RCT
325 by the end of February (=cut off for inclusion in current study)

13. Materials – iCBT programs
 Space from Depression
 Space from Anxiety
 Space from Depression
and Anxiety
 Interactive, media-rich
content
 Content and structure is
based on CBT principles
 Includes information, videos,
quizzes, interactive activities,
homework suggestion and
personal stories
 Regular reviews from trained
clinicians

14. Measures
Measure Assessment Time of assessment
Patient Health Questionnaire 9-item scale
(PHQ-9; Kroenke, Spitzer, & Williams, 2001; Spitzer,
Kroenke, & Williams, 1999)
Depression Baseline, post-
treatment, follow-up
Generalized Anxiety Disorder 7-item scale
(GAD-7; Spitzer, Kroenke, Williams, & Löwe, 2006)
Anxiety Baseline, post-
treatment, follow-up
Work and Social Adjustment Scale (5-items)
(WSAS; Mundt, Marks, Shear, & Greist, 2002)
Impaired
functioning
Baseline, post-
treatment, follow-up
Frequency of Thoughts and Actions Scale
(FATS; Terides et al., 2016)
CBT skills usage Follow-up

15. Outcome measures
 PHQ-9, GAD-7 & WSAS are part of minimum data set
routinely completed with clients in primary care in the UK
 Exhibit good reliability and validity
 Extensively used in research
(Department of Health, 2011)

16. FATS
 Assesses frequency with which CBT-related skills have
been used during the previous week
 Comprises four subscales
 cognitive restructuring, social interaction, rewarding
behaviours, activity scheduling
 Acceptable to high internal consistency of subscales &
full scale and sensitive to change during iCBT
(Terides et al., 2016)

17. FATS
(Terides et al., 2016)
In the past week how often did you: Not at
all
One or
two
days
Half the
days
Almost
every
day
Every
day
1 Change your thinking to be more realistic and helpful? 0 1 2 3 4
2 Reframe a negative situation into a more positive one? 0 1 2 3 4
3 Stop yourself from thinking unhelpful and unrealistic thoughts? 0 1 2 3 4
4 Talk about your day with a friend or family member? 0 1 2 3 4
5 Have a meaningful conversation with someone? 0 1 2 3 4
6 Talk with a friend or family member on the phone? 0 1 2 3 4
7 Work on a project that was meaningful to you? 0 1 2 3 4
8 Do something that was very satisfying to you? 0 1 2 3 4
9 Do a hoppy or personal interest on your own? 0 1 2 3 4
10 Aim to spend time with positive people? 0 1 2 3 4
11 Plan a pleasant activity to make you feel better? 0 1 2 3 4
12 Plan to do something to motivate yourself? 0 1 2 3 4
FATS total score

18. FATS – qualitative extension
 “1. As part of your SilverCloud program you were presented
with techniques and strategies (i.e. relaxation, challenging
your thoughts etc.) designed to help you cope with your
mental health difficulties better and improve your mental
wellbeing. However, different people may have experienced
these techniques and strategies differently. Are there any
particular techniques or strategies you used recently? And if
so, how have you found using them?”
 “2. Are there any techniques or strategies you were using but
have stopped using recently? If so can you tell us what kind
of techniques or strategies they were and why you stopped
using them?”

19. Data Analysis
Follow-up
outcomes
Repeated measures ANOVAs with time as independent and PHQ-9, GAD-7 and
WSAS as dependent variables
CBT skills
usage at
follow-up
Cronbach’s Alpha & descriptive statistics for the FATS
Descriptive and Interpretative analysis following Elliot and Timulak’s (2005) steps
1) data preparation
2) delineation of meaning units
3) development of organising structure of data (i.e. domains)
4) generation of categories within domains & categorisation of meaning units
5) abstracting of main findings
6) integration with quantitative results and validity checks
Mediation
analysis
Multiple regression with pre-symptom levels and CBT skills usage (i.e. FATS) as
predictors of maintenance of effect on the PHQ-9, GAD-7 & WSAS
Multiple regression with pre- and post-treatment symptom levels as predictors of
follow-up CBT skills usage to address issues around temporality in the mediation
analysis

20. Results
 Age range 18 to 74 with mean age 34
 54 female – 25 male
 31.7% - Space from Anxiety, 12.7% Space from
Depression, 55.7% Space from Depression and Anxiety
 Two participants did not log-on post-randomisation and
where therefore excluded from the study

21. Follow-up Outcomes
 Friedman’s nonparametric ANOVA was conducted
 Listwise deletion applied of two participants that did not
complete all measures  N=75
 Significant difference in depressive symptoms, anxiety and
functioning across the three time-points
 Wilcoxon tests were used to follow-up this finding
 Depressive symptoms, anxiety and functioning significantly
improved from pre- to post-treatment & remained stable at
follow-up

22. Follow-up Outcomes
Means and standard deviations across the three time-
points and effect sizes for within-group effects (N=75)
Pre-treatment
M(SD)
Post-treatment
M(SD)
Follow-up
M(SD)
Pre-treatment/
Post-treatment
effect size*
Pre-treatment/
Follow-up
effect size*
PHQ-9 12.67(5.36) 8.17(5.42) 8.03(5.84) r = -.50 r = -.50
GAD-7 11.96(5.16) 7.16(5.31) 7.16(5.30) r = -.51 r = -.51
WSAS 16.25(7.48) 11.80(8.04) 11.05(8.72) r = -.34 r = -.41

23. Follow-up Outcomes
Figure 1. Means
of PHQ-9, GAD-7
and WSAS across
three time-points
(N=75)
0
2
4
6
8
10
12
14
16
18
Pre-treatment Post-treatment 3-month follow-up
Meanscore
WSAS PHQ-9 GAD-7

24. CBT skills usage
 FATS – quantitative analysis N=77
 FATS extension – qualitative analysis  N=35 as
completion of qualitative questions was not compulsory
 There were no statistically significant differences
between the qualitative subsample and full sample

25. CBT skills usage
Reliability analysis and descriptive statistics for FATS full scale
and subscales (N=77)
Cronbach’s Alpha M(SD)
Full scale .84 22.67 (8.52)
Cognitive Restructuring .76 5.53 (2.33)
Social Interaction .74 6.77 (3.30)
Rewarding Behaviour .68 4.88 (2.71)
Activity Scheduling .80 5.49 (2.96)

26. CBT skills usage
 Qualitative analysis followed Elliot & Timulak’s (2005) steps for
descriptive and interpretive analysis
 Data was analysed within four domains
1. CBT-related techniques and strategies used
2. Experiences using techniques/strategies
3. Discontinued CBT-related techniques and strategies
4. Experiences around discontinued techniques/strategies
 Several categories & subcategories emerged within domains

27. CBT skills usage
CBT related techniques and strategies used at 3-month follow-up and participants’
experiences using them (N=35)
Techniques and strategies used NO Experiences using techniques and strategies NO
Cognitive techniques/strategies
‘Worry tree’ exercise
‘Worry time’ strategy
Thought monitoring
Challenging/changing thoughts
Self-analysis and thought-feeling-behaviour
cycles
Behavioural techniques/strategies
Breathing exercises
Relaxation exercises
Mindfulness and mediation exercises
Taking "me time"
Social support
Activation and activities
Specific goal-oriented strategies (i.e. sleep
hygiene, goal setting, problem-solving)
23
7
8
8
12
6
24
6
5
6
3
4
8
3
Helpful experience of
fostered insight and flexibility
self-compassion and self-efficacy
letting go and reduced symptoms
coping and problem solving
calm and relaxation
Hindering experiences of
insufficient effectiveness of
techniques/strategies
issues in applying techniques/ strategies in
challenging life circumstances
Proactive and on-going engagement in
selecting, tailoring and practicing
techniques/strategies
12
3
3
3
5
2
7
6
2
9

28. CBT skills usage
Proactive and on-going engagement in selecting,
tailoring and practicing techniques/strategies:
“I have found that writing the worries down and putting
them away until later mostly works for me. Some of the
time this doesn't work and then I try distracting myself”
(Pp76)
“Those are the main two that I use and have integrated
into my daily routine. I may revisit some others but I feel
that the ones I use most appropriately address the issues
that were most prominent” (Pp51)

29. CBT skills usage
CBT related techniques and strategies participants had stopped
using at 3-month follow-up and their experiences discontinuing to
use them (N = 9)
Discontinued techniques and strategies NO Experiences around discontinued
techniques and strategies
NO
Techniques around worry (i.e. "worry
time")
Breathing and relaxation exercises
Meditation
Positive action strategy (i.e. exercise)
Mood board exercise
Ignoring thoughts
3
1
1
2
1
1
Difficulties keeping up application of
techniques/strategies
Experiences of techniques/strategies
as ineffective
Experiences of other
techniques/strategies as superior
3
2
3

30. Integration of quantitative
and qualitative findings
 Data complemented each other in extreme cases
 Discrepancies in relation to citing of cognitive
restructuring and social interaction as a CBT skills in
qualitative and quantitative accounts
 Reasons for this may relate to proceduralization of skills or
varying types of social interaction (structural vs.
functional)

31. CBT skills usage &
maintenace of outcomes
 Hierarchical multiple regression with pre-treatment
symptom levels & CBT skills usage as predictors of
maintenance of effects at follow-up
 Maintenance of effect variables were calculated by
subtracting follow-up score from post-treatment score
 + denotes further improvement
 0 denotes maintenance of effect
 - denotes deterioration

32. PHQ-9 – mediation analysis
PHQ-9 multiple hierarchical regression coefficients
B SE B β
Step 1 (Constant) 1.41 1.03
PHQ-9 Pre-treatment -0.09 0.07 -0.14
Step 2 (Constant) 0.40 1.56
PHQ-9 Pre-treatment -0.08 0.07 -0.13
FATS full scale 0.04 0.05 0.10
Note R2 =.02 at step 1, R2 =.03 at step 2; p>.05 for step 1 & 2

33. GAD-7 – mediation analysis
GAD-7 multiple hierarchical regression coefficients
B SE B β
Step 1 (Constant) 1.83 1.17
GAD-7 Pre-treatment -0.07 0.09 -0.09
Step 2 (Constant) 1.79 1.74
GAD-7 Pre-treatment -0.07 0.09 -0.09
FATS full scale 0.00 0.05 0.00
Note R 2=.01 at step 1, R2 =.01 at step 2; p>.05 for step1 & 2

34. WSAS – mediation analysis
 In line with Tabachnick and Fidell‘s (2013) outlier with
z-score of -3.44 excluded to improve normality &
goodness of fit
 In hierarchical multiple regression only CBT skills usage
significantly predicted maintenance of effects
 Final model was a simple regression model

35. WSAS – mediation analysis
WSAS simple regression coefficients
Note R2 =.08; *p<.05
B SE B β
(Constant) -2.70 1.47
FATS full scale 0.16 0.06 0.29*
Model predicted 8% (adjusted 7%)of variance in how well
functioning effects were maintained

36. WSAS – mediation analysis
Figure 2. Regression plot
illustrating a linear
relationship between CBT
skills usage (as measured
by the FATS) and
maintenance of
functioning effects of iCBT
(as measured by the WSAS)
i.e. greater usage of CBT
skills predicted greater
maintenance of functional
improvements

37. Temporality in mediation
model
 In order to address issues around temporality in the
mediation model, another regression model was built to
assess potential predictors of CBT skills usage
 PHQ-9, GAD-7 & WSAS pre- & post-treatment did not
predict CBT skills usage in stepwise (backwards) multiple
regression model

38. Discussion
 In line with previous research, iCBT effective for depression &
anxiety in this study (i.e. Andersson et al., 2013; Fogliati et al., 2016; Hedman et
al., 2013)
 CBT skills usage & experiences thereof varied between
participants at follow-up
 Findings resemble helpful & hindering events users report during iCBT
(Richards & Timulak, 2012)
 Hypothesis of the study partially supported
 Contradicting previous findings of therapy-related skills predicting
depressive outcomes then (Powers, Thompson & Gallagher-Thompson, 2008)

39. Limitations
 Issues around the measurement of CBT skills (timing,
single measurement, validity of FATS, perceived
helpfulness of skills not measured)
 Issues around 8-week post-treatment measurement in a
naturalistic setting

40. Conclusion
 Current study represents one building block on the road
to establishing how people change or maintain their
outcomes after iCBT
 Main contribution lies in the hypotheses for future
research it has helped to generate

41. Future research may
explore/address…
1. the concept of social support/interaction as a CBT skill and
how it is best communicated as a CBT skill to iCBT users
2. the role of proceduralization of CBT skills vs self-responsible,
proactive use of skills
3. short-comings of the current study by including repeated
measures of CBT skills usage & perceived helpfulness thereof
4. CBT skills usage in high versus low functioning individuals as
well as whether one group benefits more from CBT skills
usage than the other

42. References
 Andersson, G. (2016). Internet-delivered psychological treatments. Annual Review of Clinical Psychology, 12, 157-179.
https://doi.org/10.1146/annurev-clinpsy-021815-093006
 Andersson, G., Hesser, H., Veilord, A., Svedling, L., Andersson, F., Sleman, O., ... Carlbring, P. (2013). Randomised
controlled non-inferiority trial with 3-year follow-up of internet-delivered versus face-to-face group cognitive
behavioural therapy for depression. Journal Of Affective Disorders, 151(3), 986-994.
https://doi.org/10.1016/j.jad.2013.08.022
 Department of Health (2011). The IAPT Data Handbook – Appendices. Retrieved from
http://serene.me.uk/articles/iapt-data-handbook-appendices.pdf
 Elliott, R. & Timulak, L. (2005). Descriptive and interpretive approaches to qualitative research. In J. Miles & P. Gilbert, A
handbook of research methods for clinical and health psychology (1st ed.). New York, NY: Oxford University Press.
 Fogliati, V., Dear, B., Staples, L., Terides, M., Sheehan, J., Johnston, L., ... Titov, N. (2016). Disorder-specific versus
transdiagnostic and clinician-guided versus self-guided internet-delivered treatment for panic disorder and comorbid
disorders: A randomized controlled trial. Journal Of Anxiety Disorders, 39, 88-102.
https://doi.org/10.1016/j.janxdis.2016.03.005
 Forand, N., Barnett, J., Strunk, D., Hindiyeh, M., Feinberg, J., & Keefe, J. (2017). Efficacy of Guided iCBT for Depression
and Mediation of Change by Cognitive Skill Acquisition. Behavior Therapy. Advance online publication.
https://doi.org/10.1016/j.beth.2017.04.004
 French, L. R., Thomas, L., Campbell, J., Kuyken, W., Lewis, G., Williams, C., ... Turner, K. M. (2017). Individuals’ Long Term
Use of Cognitive Behavioural Skills to Manage their Depression: A Qualitative Study. Behavioural and Cognitive
Psychotherapy, 45(1), 46-57. https://doi.org/10.1017/S1352465816000382

43. References
 Halmetoja, C., Malmquist, A., Carlbring, P., & Andersson, G. (2014). Experiences of internet-delivered cognitive
behavior therapy for social anxiety disorder four years later: A qualitative study. Internet Interventions, 1(3), 158-163.
https://doi.org/10.1016/j.invent.2014.08.001
 Hedman, E., Ljótsson, B., Rück, C., Bergström, J., Andersson, G., Kaldo, V., … Lindefors, N. (2013a). Effectiveness of
Internet-based cognitive behaviour therapy for panic disorder in routine psychiatric care. Acta Psychiatrica
Scandinavica, 128(6), 457-467. https://doi.org/10.1111/acps.12079
 Kroenke, K., Spitzer, R., & Williams, J. (2001). The PHQ-9. Journal Of General Internal Medicine, 16(9), 606-613.
https://doi.org/10.1046/j.1525-1497.2001.016009606.x
 Mogoașe, C., Cobeanu, O., David, O., Giosan, C., & Szentagotai, A. (2016). Internet-Based Psychotherapy for Adult
Depression: What About the Mechanisms of Change?. Journal Of Clinical Psychology, 73(1), 5-64.
https://doi.org/10.1002/jclp.22326
 Mundt, J., Marks, I., Shear, M., & Greist, J. (2002). The Work and Social Adjustment Scale: a simple measure of
impairment in functioning. British Journal Of Psychiatry, 180(5), 461-464. https://doi/10.1192/bjp.180.5.461
 Păsărelu, C., Andersson, G., Bergman Nordgren, L., & Dobrean, A. (2017). Internet-delivered transdiagnostic and
tailored cognitive behavioral therapy for anxiety and depression: a systematic review and meta-analysis of
randomized controlled trials. Cognitive Behaviour Therapy, 46(1), 1-28. https://doi.org/10.1080/16506073.2016.1231219

44. References
 Powers, D., Thompson, L., & Gallagher-Thompson, D. (2008). The Benefits of Using Psychotherapy Skills Following
Treatment for Depression: An Examination of “Afterwork” and a Test of the Skills Hypothesis in Older Adults. Cognitive
And Behavioral Practice, 15(2), 194-202. https://doi.org/10.1016/j.cbpra.2007.01.002
 Richards, D., Duffy, D., Blackburn, B., Earley, C., Enrique, A., & Palacios, J., … Timulak, L. (2018). Digital IAPT: the
effectiveness & cost-effectiveness of internet-delivered interventions for depression and anxiety disorders in the
Improving Access to Psychological Therapies programme: study protocol for a randomised control trial. BMC
Psychiatry, 18(1), 59. https://doi.org/10.1186/s12888-018-1639-5
 Richards, D., Richardson, T., Timulak, L., & McElvaney, J. (2015). The efficacy of internet-delivered treatment for
generalized anxiety disorder: A systematic review and meta-analysis. Internet Interventions, 2(3), 272-282.
https://doi.org/10.1016/j.invent.2015.07.003
 Richards, D., & Timulak, L. (2012). Client-identified helpful and hindering events in therapist-delivered vs. self-
administered online cognitive-behavioural treatments for depression in college students. Counselling Psychology
Quarterly, 25(3), 251-262. https://doi.org/10.1080/09515070.2012.703129
 Spitzer, R., Kroenke, K., Williams, J., & Löwe, B. (2006). A Brief Measure for Assessing Generalized Anxiety Disorder.
Archives Of Internal Medicine, 166(10), 1092. https://doi.org/10.1001/archinte.166.10.1092
 Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics. Boston, MA: Pearson