This document summarizes the evaluation of an e-learning training program for VA mental health providers on the Collaborative Assessment and Management of Suicidality (CAMS) approach. The study aimed to develop an online version of the in-person CAMS training and test its effectiveness compared to the in-person and non-intervention control groups. It involved multiple VA sites and over 200 providers. Developing the e-learning involved iterative revisions based on feedback. Providers were randomly assigned to conditions. Surveys measured competence, reactions, beliefs, and practice pre, post, and follow up. Focus groups provided feedback on experiences. Preliminary findings showed similar completion rates between conditions. Satisfaction surveys

1. Evaluation of Multisite E-learning
Training for VA Mental Health
Providers within the CAMS Study
1

2. Presentation
Jan York, PhD, APRN, FAAN (CO-I)1,2,3
Nurse Researcher, Research Professor
UNM Visiting Faculty
Elizabeth Marshall, MD, MBA1,3
Research Coordinator, Designer
Kathryn Magruder, PhD, MPH (PI)1,3
Research Health Scientist, Professor
Mark De Santis, PsyD, (CO-I) 1
Suicide Prevention Coordinator &Lead
Derik Yeager, MBS 1, 3
Research Associate
Ralph H. Johnson VAMC, Charleston
Affiliation
1. Ralph H. Johnson VAMC
2. UNM College of Nursing, Albuquerque, NM
3. Medical University of South Carolina
2

3. Colleagues & e-Learning Co-authors
3
Affiliations
1. The Catholic University of
America, Washington, DC
2. Ralph H. Johnson Veteran’s Administration
Medical Center, Charleston, SC
3. Medical University of South
Carolina, Charleston, SC
David A. Jobes, PhD, ABPP 1
Professor, Co-Director of Clinical Training, Co-I
Rebecca Knapp, PhD 3
Statistician, Professor
Louisa Burriss, PhD 2
Research Coordinator
Mary Mauldin, EdD 3
Professor
Stan Sulkowski, BS3
CARC Assistant
Charlene Pope, PhD, MPH, 2 ,3
Co-I, Associate Professor, Assoc Nurse
Jonathan Coultas, BA3
CARC Assistant

4. Presentation Outline
• Grant information
• Background
– Targeted intervention
– On line training
• Presentation objectives
• Methodology
• Approval
• Implementation-development, CEU’s, sites, recruitment,
delivery
• Evaluation-measures, initial findings
• Preliminary conclusions
• Next steps and off shoots
4

5. Patient and Provider Outcomes of
e-Learning Training in CAMS
Objective:
to develop and test the effectiveness
of an electronic learning alternative
to the Collaborative Assessment and
Management of Suicidality (CAMS)
in-person approach.
VA HSR&D EDU 08-424 funded
health education research
3 year (2009-2012), multisite
study, SE VISN (VA region)
5

6. Suicide in the U.S.
(2010 CDC data)
Suicide is the Tenth leading cause of death:
 >38,000 suicides that year in the
U.S. (rate 12.4)
 91 suicides occur each day
 A suicide every 13.7 minutes
 NM ranks 5th rate 20.1 (n-413)
 2 victims of homicide/3 suicides
 Twice as many deaths due to
suicide than due to HIV/AIDS
 Highest rates in elderly, midlife
men & young, males, firearms
 Almost 1 mil attempts per year, 25
attempts/ 1 completion (youth
100-200, elderly 4/1)
6

7. Background:
Veterans are at high risk for suicide
The VA has identified suicide in Veterans as
a priority.
The risk for suicide in Veterans is:
- higher than for non-Veterans
- higher for some subgroups
- higher for rural than urban Veterans
- highest in subgroups of Veterans with
depression, psychiatric treatment
The risk in military populations is highest in
the Army and the Marines.
7

8. Background: Consider a
VA- specific study of suicide
Retrospective review,887,859 Veterans
receiving depression intervention in VA
medical centers, found:
Significantly elevated rates of suicide:
- 48 weeks after hospitalization
- 12 weeks after hospitalization for
61-80 year olds (highest suicide
rate group)
- 12 weeks after medication
changes
(Valenstein et al. )
8

9. Targeted Intervention:
CAMS
The Collaborative Assessment and Management
of Suicidality (CAMS) is an overall process of
clinical assessment, treatment planning, and
management of suicidal risk.
The CAMS core multipurpose risk assessment
tool is the Suicide Status Form (SSF).
The SSF serves as a roadmap for guiding the
clinician and patient, providing crucial and
comprehensive documentation.
9

10. Suicide Status Form
The Suicide Status Form
(SSF) document is used for:
1. Assessment
2. Treatment Planning
3. Tracking
4. Outcomes
VA purchased rights to
use in CPRS
• template in process
10

11. CAMS is Consistent
with…
VA Suicide Prevention Strategy
(2009)
VISN7 & 2 CoE priorities
DOD, VA, NIMH systematic reviews
Blue ribbon panel
National and VA Recovery
Initiatives
2012 National Strategy for Suicide
Prevention (DHHS)
Objective 7 training providers in
assessment and management
11

12. Empirical Support for
CAMS
CAMS is used in multiple settings
Core SSF assessment aspects &
quantitative properties
established, qualitative support
6 published correlational studies
over 25 years supporting feasibility
& clinical use of CAMS & SSF with
suicidal outpatients & 1 inpatient
psychiatric study
Level of evidence-Best Practice
12

13. Why is training important?
A patient’s ambivalence about dying is an
opportunity for a provider to save a life.
A systematic method of managing suicidality
can assuage the fear of losing a patient.
Training can help increase confidence and
competence and dispel common myths.
13

14. Why should I use CAMS?
David Koerner, MSW,
VA provider/ champion/ early adopter of
CAMS
“I have always considered it a privilege to be
allowed into the life of an individual in crisis.
For me, one of the most challenging clients is
the person who can no longer find a reason
to live. Personal experience has shown that
this is a life threatening situation. I have
found the CAMS approach, (and specifically
the SSF tool), to be effective at engaging
suicidal persons and eliciting important
information that might help in their
recovery”.
14

15. Background:
Health Education Research
U.S. Department of Education
(2009) meta-analysis:
The effectiveness of eLearning
with blended learning compared
favorably, and generally led to
more learning than traditional
face-to-face interaction.
Mixed studies but little research
evidence for changes in practice
15

16. Background:
Systematic Reviews of Online
Education
Cook et al. 2008 in JAMA
Exemplary systematic review
51 trials (30 RCT’s) 1990-2008
compared web-based with other
educational activity or no
intervention
Findings
Large heterogeneity-
configuration, blending, presentati
on, methods
Mixed, none or insignificant
difference favoring web-based
16

17. Background: Systematic Review of
empirically-supported instructional
methods in online education
Cook et al. 2010a, 10b
Instrument measures in 266
studies
Interactivity, practice exercises,
feedback, repetition
associated with improved
learning outcomes
Interactivity, online discussion
& audio associated with
improved satisfaction
17

18. Presentation Objectives
Describe the process and outcomes
related to aims:
1) Develop CAMS e-learning including
the same material & objectives of
In-person training
2) Testing effectiveness of the e-
learning compared to in-person &
non-intervention control in terms of
provider evaluation of training
Describe offshoots of project
18

19. Methodology
• Multicenter, randomized, cluster
three group design, noninferiority
• Multivariable modeling strategy to
analyze change in
confidence, beliefs, and practice
• Pilot delivery to assess provider
evaluation and improve training
• Formative evaluation of facilitating
and inhibiting factors of the process
19

20. Approval
• IRB Medical University of
South Carolina/VA
• VA Office of Research
• Site specific VA IRBs
In hindsight - We WISH we
used VA Central IRB
20

21. Benefits of
Participation
• ARMs 1-2: 2 Training Deliveries
– CAMS Training
– 6.5 hours of CEU credit
– biweekly telephone coaching calls
– CAMS manual
• ARM 3: Control
– Emergencies in Mental Health
Practice book
21

22. Risks of Participation
May experience:
• discomfort due to
content
• increased anxiety
due to performing
new interventions
and review of patient
records
Confidentiality – risk in
all studies
22

23. Participant Eligibility
Outpatient mental health
providers-
psychiatrist, psychologist,
APRN, social
worker, case managers
(included RN’s)
No previous CAMS training
Informed consent 23

24. Implementation Time-Line
24

25. Empirically-Supported Instructional Methods
(Cook et al., 2010; Kok et al., 2004; Means et al, 2009; Williams et al., 20011)
• Evidence-based intervention strategies
• Interactivity and engagement (video clips)
• Practice exercises (patient cases) and repetition
• Blending
• Behavioral journalism
• Computer tailoring
• Simplicity, ease of use
• Organization by modules
• 24/7 accessibility and platform-independence
• Anonymous and self-paced
• Visual attractiveness and appeal
• Individuation
• Resources for help and feedback
• Instructor-direction
• Auditory information modeling
25

26. eLearning Development
Iterative process with multiple paths
and revisions
Early stages…
• In-person CAMS and Moodle (platform)
trainings for study staff
• Balancing CAMS research & “How to do
CAMS”
– Transcripts of In-person training
– Use of Jobes (2006) manual to inform
curriculum
• Guidance of education and technology
experts (development of modules, Moodle
capacity, use of web site)
26

27. e-learning Development
Video segments
27

28. 28
Example Veteran-specific Vignette

29. eLearning Development
Production stages…
• Development of scripts for
main video & 2 vignettes
reflecting diversity & short
introductions
• One day filming of Dave
Jobes and Keith Jennings
Barrier
Delivery in first site
underscored problems and
limitations
29

30. eLearning Development
Late stages…
Major revision of
eLearning curriculum
Ensuring simplicity
and adding artistic
appeal
30

31. Barriers in Development
• Microphone problems
during filming
– Subtitles developed
• Technology issues with
bandwidth
– Multiple compression
attempts in order for videos
to download
– Consultation with VISN
technology group
31

32. Barriers in Development
Limits of file sharing
• Large amount of file
graphics & security issues
(burning of DVD’s, thumb
drives)
Development of dedicated
share drive
32

33. Barriers in Development
Remember:
Great Minds Don’t
Always Think Alike!
Multisite-culture, IRB
Multidiscipline-unions
Travel to sites for training
33

34. In-Person vs. e-Learning
Both: 6.5 CEU’s
the Suicide Status Form (SSF)
The CAMS Approach to Suicide Risk Assessment
CAMS Intervention (Problem-Focused Treatment)
in-Person:
AM & PM sessions
CAMS research studies
CAMS in college population and military
Ethics/Malpractice and Next Steps
E-Learning:
Veteran specific
4 modules
CAMS video segments with Veterans
Videos illustrating Veteran diversity
VA Suicide Prevention Strategy module
34

35. Tick-Tock
Gross underestimate of
time for eLearning
development :
• Projected- 6-12 months
• Actual- 15 months
Reality of chart
abstraction-permissions,
complexity, time
No cost extension
35

36. Dissemination
Barriers- CEU’s
VA approved In-Person
CAMS brochure
New & unclear process
for e-learning
Guidelines changed in
process
Change in personnel at
TMS
36

37. Dissemination
Barriers- Websites
CAMS eLearning training
• Process for VA platform
delivery lengthy
• Website independent of VA
E-Learning CEU accreditation
on TMS website VA
• VA Training Evaluation
satisfaction
• eLearning Quiz (Social
Workers have strictest
requirements)
37

38. Provider Recruitment
& Randomization
Get Their Attention!
Goal 268 providers
309 eligible
230 (77%) consented
Completed presurvey to be
randomized (220)
75-e-leaning, 71-in-person,
76-control
IRB requirement in one site
verbal consent
38

39. Mother Nature
April 27th 2011 tornado hit
Tuscaloosa County,
Alabama
43 died & >1000
injured
Tuscaloosa VAMC served
as a morgue
People living in hotels
CAMS in person cancelled
39

40. Delivery of Training
Clinic blocking 6-8 weeks in
advance
4 In-person trainings
– Tuscaloosa attended another
site
– CHS staff attended each training
E-Learning delivery
– Available same day as in-person
– 3 week accessibility extended
40

41. Delivery:
Coaching Component
The Purpose:
Determine CAMS implementation
& increase dissemination
The Format: VANTS call with Dr.
Jobes
• 6 Bi-monthly hour sessions
(lunch & learn)
• Multiple email reminders
78 % had NO attendees
41

42. Learning Measures
• CAMS Training Surveys
– Pre-training
– Post-training
– 3 month Follow-up
• Measures 10-15 minutes
(Adapted from Jobes, Knox & VISN2 CoE)
42

43. CAMS Survey Items
Eleven Items
– Competence
– Reactions
– Beliefs
– Motivations
– Practice & CAMS
• Delivery mode-
satisfaction &
preference
• Demographics
43

44. Adoption factors:
Focus Groups
Provider experience……
• Impression of training
experience
• Experience in delivery
• Organizational incentives &
rewards for implementation
• Communication of related
organizational goals
• Organizational & facilitating
factors or barriers
• Implementation success
• Compatibility with professional
beliefs, values and practices
• Fit with workflow and program
44

45. Providers were primarily midlevel, females, 40-
49 years
Rates of completion of training similar between
training conditions and sites (one lower)
75% rate of completion of training (3
modules/sign out) by discipline
Psychologists highest rate
Provider’s career experience with suicidal
patients
32% lost > 1 patient due to suicide
75% treated > 100 suicidal
patients
8% NEVER treated a suicidal patient.
Findings: Provider and Site
Profile
45

46. Lessons Learned
• Creative recruitment-walking the halls
• Identify people/site early for product
review
• Build in a formal pilot site and
participants
• Know VA technology
• Plan for unexpected-weather barriers
and site withdrawal
• Leadership support-ACOS and SPC
• Early birds more likely to complete
• Low cost-benefit ratio of coaching
46

47. Interesting Find…
47
2 Separate Focus Groups: E-learning & in
person
Participants experience -liking both
trainings, using parts of CAMS, CAMS
similarity to VA Safety Plan
Barriers-time constraints, other required
clinical protocols/processes, few patients
qualifying for CAMS, experience of being
“rusty’ due to infrequent use
Recommendations-use a dedicated
clinician, use in younger Veterans,
integrate in other protocols and units
(inpatient), and use a SharePoint resource
for training

48. Finding: Satisfaction
48
VA Evaluation of Training
• TMS changed twice during delivery
period
• TMS provided results by condition &
site
• This limited us to descriptive statistics
• Collapsed into 7 themes by consensus
of 2 raters
• Overall
• Content
• Objectives
• Job impact
• Enablers & barriers
• Logistics
• Environment

49. 7.9% 7.7%
3.2% 1.3%
88.9% 91.0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
e-learning in-person
Disagree Neutral Agree
9.1% 9.7%
3.0% 0.0%
87.9% 90.3%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
e-learning in-person
Disagree Neutral Agree
Content
The scope of the material was appropriate to my
needs.
I found the material in this learning activity to be
relevant and up-to-date.
The content was relevant to my job-related
needs.
Objectives
Overall, I fully accomplished the learning activity's
objectives.

50. 8.3% 8.1%
13.3%
21.0%
78.3%
71.0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
e-learning in-person
Disagree Neutral Agree
7.4% 4.5%
10.1%
7.7%
82.4%
87.7%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
e-learning in-person
Disagree Neutral Agree
Logistics
I obtained information on the learning activity's
logistics (i.e. date, location, time) in a timely
manner.
If you required any accommodations for a
disability, your request was addressed respectfully
and in a timely manner.
Environment
The appropriate technology was utilized to facilitate
my learning.
The training environment was conducive to my
learning
I found that the technology in this learning activity
was easy to use.
Overall, I was satisfied with the use of technology in
this learning activity.

51. 6.9%
2.1%
3.4%
4.3%
89.7%
93.6%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
e-learning in-person
Disagree Neutral Agree
6.8% 3.9%
5.7%
3.9%
87.5%
92.2%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
e-learning in-person
Disagree Neutral Agree
Overall
Overall, I was satisfied with this learning activity.
I would recommend this learning activity to
others.
Learning
The learning activities and/or materials were
effective in helping me learn the content.
I learned new knowledge and skills from this learning
activity.

52. 10.0%
4.3%
15.0%
12.8%
75.0%
83.0%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
e-learning in-person
Disagree Neutral Agree
6.7%
14.3%
23.3% 11.1%
70.0%
74.6%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
e-learning in-person
Disagree Neutral Agree
Job Impact
I will be able to apply the knowledge and skills
learned to my job.
This learning activity will help improve my job
performance.
Enablers & Barriers
My manager and I set expectations for this learning
prior to attending this learning activity.
I feel competent to apply the skills/knowledge I
developed during the learning activity.
This learning activity aligns with the business
priorities and goals identified by my organization.

53. VA Evaluation of Training
What we know:
– Faculty Rating & Participant Satisfaction
positive for both (mostly agree or
strongly agree)
– Trend for In-person to be rated slightly
more positive & slightly less negative
53

54. 1. I don’t have anxiety about working with suicidal patients.
2. I am confident in my ability to successfully assess suicidal
patients.
3. I am confident in my ability to determine suicidal risk level in
patients.
4. I am confident in my ability to form a strong therapeutic
alliance with a suicidal patient.
5. I am confident that I can help motivate a patient to live.
6. I can develop an adequate safety/coping plan with patients
who are at-risk for suicide.
Strongly Disagree <-----------------> Strongly Agree
CAMS Post-Survey Adjusted Means by Training Condition
Post-Survey MeansSurvey Item

55. 7. I am not hesitant to ask a patient if s/he is suicidal.
8. I don’t believe that hospitalization is always
the best response for suicidal patients.
9. I believe that suicidal patients should take an active role
in all aspects of their own treatment.
10. I believe my current practices are sufficient to protect
me from liability in the event one of my patients should
complete suicide.
11. I am motivated to use what are considered the "best
practices" in suicide prevention even if it requires me to do
something different in my clinical practice.
* p-values from comparison of least squares
post-survey means from MEM
CAMS Post-Survey Adjusted Means by Training Condition
Post-Survey Means*Survey Item
p = 0.040
p = 0.029
p = 0.003
Strongly Disagree <-----------------> Strongly Agree

56. Conclusions-
Breaking New Ice
• The complexity of integrating
product development, training
dissemination, and evaluation of
health education
- bumpy, unpredictable road
• The gift was our multitalented
team and collaboration
• Little known about health
education research that includes
assessing patient outcomes
56

57. Conclusions-
Breaking New Ice
CAMS eLearning appears to
be as effective as CAMS
in-person learning
57

58. Next Steps
Patient Level Analyses…
• 3 Month Survey Follow
up analysis
• Assessing high risk flag
patient (>300)outcomes
and provider adherence
for one year post training
(>10 page abstraction form)
58

59. Next Steps
Dissemination
• National VA Cyber
Seminar Feb 2012
• TMS & DOD invitations to
offer training nationally
• DOD Suicide Prevention
Workshop June 2012
• VA Nursing Research Day
59

60. Off Shoots & Next
Steps
• Systems Improvement project
• Inpatient CAMS group
Training nursing staff in group interventions
• Manuscript on suicide-specific inpatient
safety in review
• Cost analysis (VA QUERI)
• Mentoring team to be first author on
manuscripts
60

61. Next Steps
Manuscript, Online
Development and Evaluation
Recommendations
• Use intervention mapping for
development
• Develop competencies for CAMS
• Use Gorrindo Measure of System
Usability
• Allow time for iterative process
and barriers
61

62. Articles

63. Articles

64. References
Bagley S, Munjas B, Shekelle P. A systematic review of suicide prevention
programs for military or Veterans. Suicide and Life-Threatening Behavior
2010; 40:257-265.
Bossarte R, Claassen C, Knox K. Veteran suicide prevention: emerging
priorities and opportunities for intervention. Military Medicine 2010;
175:461462.
Brenner L, Department of Veterans Affairs, Centers for Disease Control and
Prevention, Department of Defense. Self-directed Violence (SDV)
Classification System. 2010.
Department of Veterans Affairs, Health Services Research and Development
Services. Strategies for Suicide Prevention in Veterans. Washington DC:
Department of Veterans Affairs; January 2009.
Department of Veterans Affairs. Office of inspector general implementing
VHA’s mental health strategic plan initiatives for suicide prevention. 2009.
http://www.va.gov/oig/publications/reports-list.asp. Accessed July 29,
2009.
Hawks S, Smith T. Thomas H, et al. The forgotten dimensions in health
education research. Health Education Research 2008; 23:319-324.
Jobes D. Managing Suicidal Risk: A Collaborative Approach. New York, NY:
Guilford Press; 2006.
Jobes D, Comtois K, Brenner L, Gutierrez P. Clinical Trial Feasibility Studies of
the Collaborative Assessment and Management of Suicidality (CAMS). In
R O’Connor, S Platt, J Gordon (eds), International Handbook of Suicide
Prevention: Research, Policy & Practice. Chichester, UK, Wiley –Blackwell:
2011.
64

65. References
Magruder K, York J, Jobes D, et al. Patient and provider outcomes of e-learning training
in CAMS. EDU 08-424.Health Services R &D, Department of Veterans Affairs.
8/1/09-7/31/12.
Means B, Toyama Y, Murphy R, Bakia M, Jones K. Evaluation of evidence-based
practices in online learning: A meta-analysis and review of online studies. U.S.
Center for Technology in Learning, Office of Planning, Evaluation, and Policy
Development, U.S. Department of Education 2009. Available at:
http://www.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf.
Accessed on January 12, 2012.
Oordt M, Jobes D, Fonseca V, et al. Training mental health professionals to assess and
manage suicidal behavior: Can provider confidence and practice behaviors be
altered. Suicide and Life-Threatening Behavior 2009; 39:21-32.
Report of the Blue Ribbon Work Group on Suicide Prevention in the Veteran
Population. www.
mentalhealth.va.gov/suicide_prevention/Blue_Ribbon_Report_FINAL_June30_08.
pdf.
Seal K, Bertenthal D, Miner C et al. Bringing the war back home: Mental health
disorders among 103,788 US Veterans returning from Iraq and Afghanistan seen at
Department of Veterans Affairs Facilities. Archives of Internal Medicine 2007;
167:476-82.
Sundararaman R, Panangala S, Lister S. Among Veterans- CRS Report to Congress Report
for Congress. Washington, DC: Congressional Research Services, Domestic Social
Policy Division; 2008.
Valenstein M, Kim H, Ganoczy D et al. Higher-risk periods for suicide among VA patients
receiving depression treatment: Prioritizing suicide prevention efforts. Journal of
Affective Disorders 2009; 112:50-58.
Williams R, Gatien G, Haggerty B. Design element alternatives for stress-management
intervention websites. Nursing Outook 2011: 59: 286-291. 65

66. Contact Information
Jan York, PhD, APRN, FAAN (CO-I)
Nursing Researcher, UNM Visiting Faculty
MUSC Research Professor
janet.york@va.gov
Elizabeth Marshall, MD, MBA
Research Coordinator, Designer
elizabeth.marshall3@va.gov
Kathryn Magruder, PhD, MPH (PI)
Research Health Scientist
MUSC Professor
magrudkm@musc.edu
66