York, janet school day presentation research


Published on

Published in: Health & Medicine, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Good afternoonIt is exciting to be here and be with you in-personThis is my second term as College Visiting FacultyI am very impressed with the studentsI have been fortunate to be mentored by Dr. Averill and Dr. Parshall and am learning so much in many spheresMy story of moving to online education
  • I am Jan York. My colleagues and I have given parts of this presentation for a national VA cyber seminar & VA/DOD meetingI am a child psych APRN and family therapist so I am all about building teams and mentoring the next generation
  • This is definitely a team effort and we want to acknowledge all our colleagues in the study. Our VA and MUSC have a strong connection and are adjacent. We have a VA Nursing Academy. UNM has applied for one. Especially Dr. Dave Jobes the developerDave and I have both been suicidologists since the 80’s.We have a strong nursing imprint on this studyI wrote a concept paper before I came to VANA which allowed us to submit the grantOur own Dr. Pope (Nse Exec for Research at the VA) is our qualitative expert and is a mentor and an inspiration for meAnd my office mate Dr. Sternke is our data abstractor and is pulling her hair out as she wades through CPRSDr. Louisa Burris is a nurse psychologist who worked in the study
  • Iplan to present information on the grant, some background, objectives, methods, implementation, evaluation, preliminary conclusions, and next steps. I will end with our off shoot.
  • So this is the grant title The project is health education researchThe Overall aim is the development and comparison of in-person and eLearning versions of the Collaborative Assessment and Management of Suicidality, hereafter referred to as CAMS A 3 year multisite study. In a few slides I will go over the project objectives that we are covering in this presentation Something you may not be aware of is VA grants don’t give salary money if you are already a VA employee
  • The data is alarmingIn 2010 we lost over 38,000 persons to suicide almost 100 per dayLet’s talk about NMranks 5thrate 1 ¾ national rateover 400 in that yearRate of homicide to suicide is 2 : 3Highest ratesAttempt 1 mil per yearThere is work to be done hereGarrett Smith grantsystematic review
  • Some backgroundSuicide prevention is a VA priorityThe risk is elevated in some subgroups and in rural VeteransIn the military the Army rate is the highest of the branches but recently decreased someI am a national trainer for the military providers in managing suicide and we just finished training over 1000 Navy & Marinemental health professional around the worldthere is data on our Air Force training several years ago
  • Interesting study by Dr. Valenstein and her colleagues of almost 900,000 Vets receiving VA care for depression They identified three risk periods for suicide
  • So we targeted CAMSCAMS is a process for assessment, planning, tracking and management of suicidal riskThere is an associated core tool-----the Suicide Status FormThe VA has purchased usage rights for the SSF’s ($40,000) and is in the process of implementing a template into CPRSWhat do think the cost is for the one primary trainer in CAMS?
  • The SSF has been validated in several studies.SSF is 8 pages published in the CAMS manualAs you can see it is used for all phases of management.How much did it cost to purchase rights for SSF use and patient record template
  • In terms of the value of this study for the VACAMS is often cited as an empirically-based best practiceA VA Blue Ribbon Panel recommended training and systematic assessment and managementAs did the recent 2012 National Strategy for Suicide prevention It is also consistent with another national and VA priority---recovery All persons with psychiatric disorder have the right to function at their highest, be involved in care and not be traumatized by their care There is a national center on recovery and the Am Psych Nurses Assoc is one of the prof orgs that will be offering an online course on recovery
  • CAMS is used in multiple settings There is empirical support for quantitative and qualitative aspects of the SSFThere are other VA and military projects in process nowOur VISN was already bringing Dr. Jobes to do training and it was very expensiveCAMS training is consistent with the VA priority to hire and train more mental health professionals
  • Why is training providers in CAMS importantA patient’s ambivalence about dying is an opportunity to save a life.It is important to train providers in a systematic method of managing suicidality to decrease their fears Confidence and competence is associated with better patient care.
  • Here is a quote from a previously trained provider who uses CAMSWe learned that using this endorsement is behavioral journalism
  • A body of literature is starting to build in health education, comparing online and in person studies.This Dept of Ed review actually showed Blended eLearning (which is eLearning and face-to-face combined) had resulted in more learning than face-to-face trainingThe studies are mixed about changes in practice.______________________Note: this is an adult study (70+ studies)
  • THIS IS MY FAVORITE WORK I LOVE SYSTEMATIC Reviews Cook published this in JAMA51 trials over 18 tears 30 RCT’sLarge differences in configuration, blending, presentation, methodsBottom line mixed, none or insignificant differences favoring online
  • So now Cook takes 266 studies (overlap) and asks the questionis there empirical support for instructional methodsWhat is associated with improved learningSatisfactiona secondary measureAny of this sound familiarHANDOUTSresearch themesdefinition of terms
  • In this presentation we are going to focus on two aims of the overall project.Describe process and preliminary outcomesVA is big on processDeveloping the e-Learning and comparing e-Learning, In person and no trainingand test effectiveness Non inferiority comes from product researchyou hope product is not sig worse as good as Describe offshoots
  • The study is a randomized cluster 3 group designNote: cluster is the delivery method (in person vs. online) stratified by disciplineMulitvariate modeling strategy to analyze change in confidence, beliefs and practiceWe will focus on online development, the pilot, implementation, provider evaluation and barriersI am intimidate by these three statisticians I have met her so I hopethey don’t ask me the hard questions
  • We had all the study approvals Central IRB was not possible at the time this was approved for funding.
  • So what did we offer provider ?If they were randomized to either training they got CEU’s, the CAMS workbook and coaching calls If randomized to no training they got the Kleeepsie book
  • Risk’s were subject matter training and changing one’s practice or observing practice can cause discomfort and anxietyVA very rigorous rules about CPRS access etc
  • So who was eligible?Trained licensed mental health providersThose notpreviously CAMS trained And those who completed informed consent
  • Suffice to say we were forced to make many revisions to our time-line.Issues of hiring, development
  • So in the development of our eCAMSWe were delighted to find empirically based practices for eLearning that have been identifiedour education expert Dr. Mary Mauldin tried to keep us on trackLater we found Cook and he is the geru
  • Developing the eLearning was an iterative journeyEarly into the project we trained staff in the CAMS intervention and the Moodle platform…that was easy partWe tried and abandoned on many different approaches before we settled into one-transcribing every word settled on very manualized versionLuckily we had the right experts to accomplish this
  • So here’s what a slide shot of our video looks likeVideo SSF printableTrnascription
  • Dave Jobes, and Keith Jennings were respectively nominated for best actor and best supporting actor
  • In the Production stagesWe developed introductions, scripts and short vignettesThe Patient was played by Keith Jennings, Keith is an Army Veteran who was deployed and lost a fellow soldier in Iraq. He is now a Catholic University student of Dave Jobes We wanted to reflect Veteran diversity by including CAMS assessments with a female and an older Vet Even though we did a pilot of 10 providersDelivery in Charleston, the first site underscored problems and limitations, now treating that as a pilot
  • In the Late stages of development…There was a Major revision of eLearning curriculum This ensured simplicity and added artistic appeal The Slides in this presentation reflect the style of our eLearning
  • There were many Barriers in DevelopmentWe encountered numerous technology problems:Microphone glitches which led to the development of subtitlesand bandwidth issues that resulted in VISN consultation and multiple video compressions
  • One of the biggest barriers at the VA was file sharing due to their sizeWe started using a dedicated share drive for all our project documents.
  • Remember, Great minds don’t always think alike! One of the biggest challenges you will encounter is the large variation of styles and views within your team and sites we had 5 sitesI happen to believe nurses are the best project coordinators other than our Dr. Marshall
  • Here’s a brief comparison of the two deliveries Both 6.5 CEU’s and include use of assessment, intervention and the SSFIn-person focuses more on CAMS research and includes studies with college students and active duty military and forensic issues We had no control over the in Person and Dr. Jobes is always changing it weaving in new research and productsThe VA eLearning is Vet specific including videos of the whole assessment session and several treatment sessions It also includes 12 completed and printable SSF forms It also includes 2 CAMS assessments reflecting diversity
  • As we all know…. Time isn’t usually on your side! Unfortunately we had grossly underestimated the development phase and data abstraction for over 300 records of the patients on the high risk flag
  • CEU’s presented a huge learning curve. Even though our TMS has many advancements… their process for eLearning placed outside of TMS development is still in it’s infancyThere were many aspects that were new to not only to US but TMS as well
  • We had multiple website challenges in order to earn CEU credits our elearners had to do the following…log-on to our delivery website, take the eLearning… then follow a link to the TMS website and take a quiz and a satisfaction survey there. Last but not least….a third website had to be accessed to take the post-survey and 3 month survey.
  • In our power analysis we needed 268 providers.There was a wide variability in the eligibility of providers and their recruitment.Our SPC, Dr. De Santis, had us walking the halls and knocking on doors to recruit.APRN’s are eager participants and adopters
  • Mother Nature…the tragedy that hit Tuscaloosa. Tuscaloosa VAMC was used primarily as a morgue.For obvious reasons….It had a grave impact on our VA’s function…and there were no hotel rooms for providers to travel from CBOC’sTtheir training was rescheduled to a later date and combined with a site
  • In order to deliver training…clinics were blocked in advance for both training groupsDr. Jobes did the 4 In-person trainings & a representative of the Charleston staff attended each trainingeLearning was available the same day as in-person (except at the initial site) providers had 2+ weeks to finish but we had to extend itEarly birds were more likely to complete
  • We were excited to offer a coaching component to our learning…The evidence shows mixed learning is more effective than In-person aloneWe offered 6 live coaching call with Dr. Jobes for both groupsDespite multiple reminders we had little participation
  • For our learning measures we used CAMS training surveys….We had adapted a CAMS survey that was developed at VISN2We used it for the pre-, the post & at a 3 month follow-up
  • The survey itself assessed factors related to practice and learning preference
  • Our co I is a great qualitative researcherShe developed a protocol to examine provider adoption
  • Then there was the Recruitment of ParticipantsThere was variability's with informed consent, one site’s IRB even allowed for verbal consent Although retrospectively we believe that this commitment may be less binding
  • We learned many lessons along the way… These are just a few
  • Synopsis of focus groups
  • Synopsis of focus groups
  • Let’s look at VA evaluation of Training and trainersItems were grouped in 8 areas-content, objectives, logistics, environment, overall satisfaction & recommend to others, learning, job impact, enablers & barriersThe VA revised this during our delivery of training5 point Likert scaleVery similar between groups in terms of primarily Agree and Strongly agree towards positive evaluation
  • We know that Dr. Jobes got high marks and providers were satisfied with in-person training on the VA evaluation
  • It is a bumpy road to bring In-person to eLearning, to recruit and get providers to complete the training and the evaluation We have been gifted with a great team with multiple talents and great collaboratorsWe are now tackling the next phaseIn regards to training mental health providers, the assessment of training on patient outcomes is not well developed
  • It is a bumpy road to bring In-person to eLearning, to recruit and get providers to complete the training and the evaluation We have been gifted with a great team with multiple talents and great collaborators like Dr. Jobes and our SPC Dr. De Santis in our journeyWe are now tackling the next phaseIn regards to training mental health providers, the assessment of training on patient outcomes is not well developed
  • This is review article on suicide in Veterans using the British Research Impact frameworkExamine research, clinical, policy and society impact in an areaPsychiatric Quarterly
  • As part of VA nursing academy asked to develop inpatient program for homeless, suicidal veterans who come in to the inpatient unitdeveloped 3 pronged approach with CAMS, recovery, and homeless intervention
  • Here are our references. We appreciate your attendance and interest in our study.
  • Here’s our contact information
  • York, janet school day presentation research

    1. 1. Evaluation of Multisite E-learningTraining for VA Mental HealthProviders within the CAMS Study1
    2. 2. PresentationJan York, PhD, APRN, FAAN (CO-I)1,2,3Nurse Researcher, Research ProfessorUNM Visiting FacultyElizabeth Marshall, MD, MBA1,3Research Coordinator, DesignerKathryn Magruder, PhD, MPH (PI)1,3Research Health Scientist, ProfessorMark De Santis, PsyD, (CO-I) 1Suicide Prevention Coordinator &LeadDerik Yeager, MBS 1, 3Research AssociateRalph H. Johnson VAMC, CharlestonAffiliation1. Ralph H. Johnson VAMC2. UNM College of Nursing, Albuquerque, NM3. Medical University of South Carolina2
    3. 3. Colleagues & e-Learning Co-authors3Affiliations1. The Catholic University ofAmerica, Washington, DC2. Ralph H. Johnson Veteran’s AdministrationMedical Center, Charleston, SC3. Medical University of SouthCarolina, Charleston, SCDavid A. Jobes, PhD, ABPP 1Professor, Co-Director of Clinical Training, Co-IRebecca Knapp, PhD 3Statistician, ProfessorLouisa Burriss, PhD 2Research CoordinatorMary Mauldin, EdD 3ProfessorStan Sulkowski, BS3CARC AssistantCharlene Pope, PhD, MPH, 2 ,3Co-I, Associate Professor, Assoc NurseJonathan Coultas, BA3CARC Assistant
    4. 4. Presentation Outline• Grant information• Background– Targeted intervention– On line training• Presentation objectives• Methodology• Approval• Implementation-development, CEU’s, sites, recruitment,delivery• Evaluation-measures, initial findings• Preliminary conclusions• Next steps and off shoots4
    5. 5. Patient and Provider Outcomes ofe-Learning Training in CAMSObjective:to develop and test the effectivenessof an electronic learning alternativeto the Collaborative Assessment andManagement of Suicidality (CAMS)in-person approach.VA HSR&D EDU 08-424 fundedhealth education research3 year (2009-2012), multisitestudy, SE VISN (VA region)5
    6. 6. Suicide in the U.S.(2010 CDC data)Suicide is the Tenth leading cause of death: >38,000 suicides that year in theU.S. (rate 12.4) 91 suicides occur each day A suicide every 13.7 minutes NM ranks 5th rate 20.1 (n-413) 2 victims of homicide/3 suicides Twice as many deaths due tosuicide than due to HIV/AIDS Highest rates in elderly, midlifemen & young, males, firearms Almost 1 mil attempts per year, 25attempts/ 1 completion (youth100-200, elderly 4/1)6
    7. 7. Background:Veterans are at high risk for suicideThe VA has identified suicide in Veterans asa priority.The risk for suicide in Veterans is:- higher than for non-Veterans- higher for some subgroups- higher for rural than urban Veterans- highest in subgroups of Veterans withdepression, psychiatric treatmentThe risk in military populations is highest inthe Army and the Marines.7
    8. 8. Background: Consider aVA- specific study of suicideRetrospective review,887,859 Veteransreceiving depression intervention in VAmedical centers, found:Significantly elevated rates of suicide:- 48 weeks after hospitalization- 12 weeks after hospitalization for61-80 year olds (highest suiciderate group)- 12 weeks after medicationchanges(Valenstein et al. )8
    9. 9. Targeted Intervention:CAMSThe Collaborative Assessment and Managementof Suicidality (CAMS) is an overall process ofclinical assessment, treatment planning, andmanagement of suicidal risk.The CAMS core multipurpose risk assessmenttool is the Suicide Status Form (SSF).The SSF serves as a roadmap for guiding theclinician and patient, providing crucial andcomprehensive documentation.9
    10. 10. Suicide Status FormThe Suicide Status Form(SSF) document is used for:1. Assessment2. Treatment Planning3. Tracking4. OutcomesVA purchased rights touse in CPRS• template in process10
    11. 11. CAMS is Consistentwith…VA Suicide Prevention Strategy(2009)VISN7 & 2 CoE prioritiesDOD, VA, NIMH systematic reviewsBlue ribbon panelNational and VA RecoveryInitiatives2012 National Strategy for SuicidePrevention (DHHS)Objective 7 training providers inassessment and management11
    12. 12. Empirical Support forCAMSCAMS is used in multiple settingsCore SSF assessment aspects &quantitative propertiesestablished, qualitative support6 published correlational studiesover 25 years supporting feasibility& clinical use of CAMS & SSF withsuicidal outpatients & 1 inpatientpsychiatric studyLevel of evidence-Best Practice12
    13. 13. Why is training important?A patient’s ambivalence about dying is anopportunity for a provider to save a life.A systematic method of managing suicidalitycan assuage the fear of losing a patient.Training can help increase confidence andcompetence and dispel common myths.13
    14. 14. Why should I use CAMS?David Koerner, MSW,VA provider/ champion/ early adopter ofCAMS“I have always considered it a privilege to beallowed into the life of an individual in crisis.For me, one of the most challenging clients isthe person who can no longer find a reasonto live. Personal experience has shown thatthis is a life threatening situation. I havefound the CAMS approach, (and specificallythe SSF tool), to be effective at engagingsuicidal persons and eliciting importantinformation that might help in theirrecovery”.14
    15. 15. Background:Health Education ResearchU.S. Department of Education(2009) meta-analysis:The effectiveness of eLearningwith blended learning comparedfavorably, and generally led tomore learning than traditionalface-to-face interaction.Mixed studies but little researchevidence for changes in practice15
    16. 16. Background:Systematic Reviews of OnlineEducationCook et al. 2008 in JAMAExemplary systematic review51 trials (30 RCT’s) 1990-2008compared web-based with othereducational activity or nointerventionFindingsLarge heterogeneity-configuration, blending, presentation, methodsMixed, none or insignificantdifference favoring web-based16
    17. 17. Background: Systematic Review ofempirically-supported instructionalmethods in online educationCook et al. 2010a, 10bInstrument measures in 266studiesInteractivity, practice exercises,feedback, repetitionassociated with improvedlearning outcomesInteractivity, online discussion& audio associated withimproved satisfaction17
    18. 18. Presentation ObjectivesDescribe the process and outcomesrelated to aims:1) Develop CAMS e-learning includingthe same material & objectives ofIn-person training2) Testing effectiveness of the e-learning compared to in-person &non-intervention control in terms ofprovider evaluation of trainingDescribe offshoots of project18
    19. 19. Methodology• Multicenter, randomized, clusterthree group design, noninferiority• Multivariable modeling strategy toanalyze change inconfidence, beliefs, and practice• Pilot delivery to assess providerevaluation and improve training• Formative evaluation of facilitatingand inhibiting factors of the process19
    20. 20. Approval• IRB Medical University ofSouth Carolina/VA• VA Office of Research• Site specific VA IRBsIn hindsight - We WISH weused VA Central IRB20
    21. 21. Benefits ofParticipation• ARMs 1-2: 2 Training Deliveries– CAMS Training– 6.5 hours of CEU credit– biweekly telephone coaching calls– CAMS manual• ARM 3: Control– Emergencies in Mental HealthPractice book21
    22. 22. Risks of ParticipationMay experience:• discomfort due tocontent• increased anxietydue to performingnew interventionsand review of patientrecordsConfidentiality – risk inall studies22
    23. 23. Participant EligibilityOutpatient mental healthproviders-psychiatrist, psychologist,APRN, socialworker, case managers(included RN’s)No previous CAMS trainingInformed consent 23
    24. 24. Implementation Time-Line24
    25. 25. Empirically-Supported Instructional Methods(Cook et al., 2010; Kok et al., 2004; Means et al, 2009; Williams et al., 20011)• Evidence-based intervention strategies• Interactivity and engagement (video clips)• Practice exercises (patient cases) and repetition• Blending• Behavioral journalism• Computer tailoring• Simplicity, ease of use• Organization by modules• 24/7 accessibility and platform-independence• Anonymous and self-paced• Visual attractiveness and appeal• Individuation• Resources for help and feedback• Instructor-direction• Auditory information modeling25
    26. 26. eLearning DevelopmentIterative process with multiple pathsand revisionsEarly stages…• In-person CAMS and Moodle (platform)trainings for study staff• Balancing CAMS research & “How to doCAMS”– Transcripts of In-person training– Use of Jobes (2006) manual to informcurriculum• Guidance of education and technologyexperts (development of modules, Moodlecapacity, use of web site)26
    27. 27. e-learning DevelopmentVideo segments27
    28. 28. 28Example Veteran-specific Vignette
    29. 29. eLearning DevelopmentProduction stages…• Development of scripts formain video & 2 vignettesreflecting diversity & shortintroductions• One day filming of DaveJobes and Keith JenningsBarrierDelivery in first siteunderscored problems andlimitations29
    30. 30. eLearning DevelopmentLate stages…Major revision ofeLearning curriculumEnsuring simplicityand adding artisticappeal30
    31. 31. Barriers in Development• Microphone problemsduring filming– Subtitles developed• Technology issues withbandwidth– Multiple compressionattempts in order for videosto download– Consultation with VISNtechnology group31
    32. 32. Barriers in DevelopmentLimits of file sharing• Large amount of filegraphics & security issues(burning of DVD’s, thumbdrives)Development of dedicatedshare drive32
    33. 33. Barriers in DevelopmentRemember:Great Minds Don’tAlways Think Alike!Multisite-culture, IRBMultidiscipline-unionsTravel to sites for training33
    34. 34. In-Person vs. e-LearningBoth: 6.5 CEU’sthe Suicide Status Form (SSF)The CAMS Approach to Suicide Risk AssessmentCAMS Intervention (Problem-Focused Treatment)in-Person:AM & PM sessionsCAMS research studiesCAMS in college population and militaryEthics/Malpractice and Next StepsE-Learning:Veteran specific4 modulesCAMS video segments with VeteransVideos illustrating Veteran diversityVA Suicide Prevention Strategy module34
    35. 35. Tick-TockGross underestimate oftime for eLearningdevelopment :• Projected- 6-12 months• Actual- 15 monthsReality of chartabstraction-permissions,complexity, timeNo cost extension35
    36. 36. DisseminationBarriers- CEU’sVA approved In-PersonCAMS brochureNew & unclear processfor e-learningGuidelines changed inprocessChange in personnel atTMS36
    37. 37. DisseminationBarriers- WebsitesCAMS eLearning training• Process for VA platformdelivery lengthy• Website independent of VAE-Learning CEU accreditationon TMS website VA• VA Training Evaluationsatisfaction• eLearning Quiz (SocialWorkers have strictestrequirements)37
    38. 38. Provider Recruitment& RandomizationGet Their Attention!Goal 268 providers309 eligible230 (77%) consentedCompleted presurvey to berandomized (220)75-e-leaning, 71-in-person,76-controlIRB requirement in one siteverbal consent38
    39. 39. Mother NatureApril 27th 2011 tornado hitTuscaloosa County,Alabama43 died & >1000injuredTuscaloosa VAMC servedas a morguePeople living in hotelsCAMS in person cancelled39
    40. 40. Delivery of TrainingClinic blocking 6-8 weeks inadvance4 In-person trainings– Tuscaloosa attended anothersite– CHS staff attended each trainingE-Learning delivery– Available same day as in-person– 3 week accessibility extended40
    41. 41. Delivery:Coaching ComponentThe Purpose:Determine CAMS implementation& increase disseminationThe Format: VANTS call with Dr.Jobes• 6 Bi-monthly hour sessions(lunch & learn)• Multiple email reminders78 % had NO attendees41
    42. 42. Learning Measures• CAMS Training Surveys– Pre-training– Post-training– 3 month Follow-up• Measures 10-15 minutes(Adapted from Jobes, Knox & VISN2 CoE)42
    43. 43. CAMS Survey ItemsEleven Items– Competence– Reactions– Beliefs– Motivations– Practice & CAMS• Delivery mode-satisfaction &preference• Demographics43
    44. 44. Adoption factors:Focus GroupsProvider experience……• Impression of trainingexperience• Experience in delivery• Organizational incentives &rewards for implementation• Communication of relatedorganizational goals• Organizational & facilitatingfactors or barriers• Implementation success• Compatibility with professionalbeliefs, values and practices• Fit with workflow and program44
    45. 45. Providers were primarily midlevel, females, 40-49 yearsRates of completion of training similar betweentraining conditions and sites (one lower)75% rate of completion of training (3modules/sign out) by disciplinePsychologists highest rateProvider’s career experience with suicidalpatients32% lost > 1 patient due to suicide75% treated > 100 suicidalpatients8% NEVER treated a suicidal patient.Findings: Provider and SiteProfile45
    46. 46. Lessons Learned• Creative recruitment-walking the halls• Identify people/site early for productreview• Build in a formal pilot site andparticipants• Know VA technology• Plan for unexpected-weather barriersand site withdrawal• Leadership support-ACOS and SPC• Early birds more likely to complete• Low cost-benefit ratio of coaching46
    47. 47. Interesting Find…472 Separate Focus Groups: E-learning & inpersonParticipants experience -liking bothtrainings, using parts of CAMS, CAMSsimilarity to VA Safety PlanBarriers-time constraints, other requiredclinical protocols/processes, few patientsqualifying for CAMS, experience of being“rusty’ due to infrequent useRecommendations-use a dedicatedclinician, use in younger Veterans,integrate in other protocols and units(inpatient), and use a SharePoint resourcefor training
    48. 48. Finding: Satisfaction48VA Evaluation of Training• TMS changed twice during deliveryperiod• TMS provided results by condition &site• This limited us to descriptive statistics• Collapsed into 7 themes by consensusof 2 raters• Overall• Content• Objectives• Job impact• Enablers & barriers• Logistics• Environment
    49. 49. 7.9% 7.7%3.2% 1.3%88.9% 91.0%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral Agree9.1% 9.7%3.0% 0.0%87.9% 90.3%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral AgreeContentThe scope of the material was appropriate to myneeds.I found the material in this learning activity to berelevant and up-to-date.The content was relevant to my job-relatedneeds.ObjectivesOverall, I fully accomplished the learning activitysobjectives.
    50. 50. 8.3% 8.1%13.3%21.0%78.3%71.0%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral Agree7.4% 4.5%10.1%7.7%82.4%87.7%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral AgreeLogisticsI obtained information on the learning activityslogistics (i.e. date, location, time) in a timelymanner.If you required any accommodations for adisability, your request was addressed respectfullyand in a timely manner.EnvironmentThe appropriate technology was utilized to facilitatemy learning.The training environment was conducive to mylearningI found that the technology in this learning activitywas easy to use.Overall, I was satisfied with the use of technology inthis learning activity.
    51. 51. 6.9%2.1%3.4%4.3%89.7%93.6%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral Agree6.8% 3.9%5.7%3.9%87.5%92.2%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral AgreeOverallOverall, I was satisfied with this learning activity.I would recommend this learning activity toothers.LearningThe learning activities and/or materials wereeffective in helping me learn the content.I learned new knowledge and skills from this learningactivity.
    52. 52. 10.0%4.3%15.0%12.8%75.0%83.0%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral Agree6.7%14.3%23.3% 11.1%70.0%74.6%0%10%20%30%40%50%60%70%80%90%100%e-learning in-personDisagree Neutral AgreeJob ImpactI will be able to apply the knowledge and skillslearned to my job.This learning activity will help improve my jobperformance.Enablers & BarriersMy manager and I set expectations for this learningprior to attending this learning activity.I feel competent to apply the skills/knowledge Ideveloped during the learning activity.This learning activity aligns with the businesspriorities and goals identified by my organization.
    53. 53. VA Evaluation of TrainingWhat we know:– Faculty Rating & Participant Satisfactionpositive for both (mostly agree orstrongly agree)– Trend for In-person to be rated slightlymore positive & slightly less negative53
    54. 54. 1. I don’t have anxiety about working with suicidal patients.2. I am confident in my ability to successfully assess suicidalpatients.3. I am confident in my ability to determine suicidal risk level inpatients.4. I am confident in my ability to form a strong therapeuticalliance with a suicidal patient.5. I am confident that I can help motivate a patient to live.6. I can develop an adequate safety/coping plan with patientswho are at-risk for suicide.Strongly Disagree <-----------------> Strongly AgreeCAMS Post-Survey Adjusted Means by Training ConditionPost-Survey MeansSurvey Item
    55. 55. 7. I am not hesitant to ask a patient if s/he is suicidal.8. I don’t believe that hospitalization is alwaysthe best response for suicidal patients.9. I believe that suicidal patients should take an active rolein all aspects of their own treatment.10. I believe my current practices are sufficient to protectme from liability in the event one of my patients shouldcomplete suicide.11. I am motivated to use what are considered the "bestpractices" in suicide prevention even if it requires me to dosomething different in my clinical practice.* p-values from comparison of least squarespost-survey means from MEMCAMS Post-Survey Adjusted Means by Training ConditionPost-Survey Means*Survey Itemp = 0.040p = 0.029p = 0.003Strongly Disagree <-----------------> Strongly Agree
    56. 56. Conclusions-Breaking New Ice• The complexity of integratingproduct development, trainingdissemination, and evaluation ofhealth education- bumpy, unpredictable road• The gift was our multitalentedteam and collaboration• Little known about healtheducation research that includesassessing patient outcomes56
    57. 57. Conclusions-Breaking New IceCAMS eLearning appears tobe as effective as CAMSin-person learning57
    58. 58. Next StepsPatient Level Analyses…• 3 Month Survey Followup analysis• Assessing high risk flagpatient (>300)outcomesand provider adherencefor one year post training(>10 page abstraction form)58
    59. 59. Next StepsDissemination• National VA CyberSeminar Feb 2012• TMS & DOD invitations tooffer training nationally• DOD Suicide PreventionWorkshop June 2012• VA Nursing Research Day59
    60. 60. Off Shoots & NextSteps• Systems Improvement project• Inpatient CAMS groupTraining nursing staff in group interventions• Manuscript on suicide-specific inpatientsafety in review• Cost analysis (VA QUERI)• Mentoring team to be first author onmanuscripts60
    61. 61. Next StepsManuscript, OnlineDevelopment and EvaluationRecommendations• Use intervention mapping fordevelopment• Develop competencies for CAMS• Use Gorrindo Measure of SystemUsability• Allow time for iterative processand barriers61
    62. 62. Articles
    63. 63. Articles
    64. 64. ReferencesBagley S, Munjas B, Shekelle P. A systematic review of suicide preventionprograms for military or Veterans. Suicide and Life-Threatening Behavior2010; 40:257-265.Bossarte R, Claassen C, Knox K. Veteran suicide prevention: emergingpriorities and opportunities for intervention. Military Medicine 2010;175:461462.Brenner L, Department of Veterans Affairs, Centers for Disease Control andPrevention, Department of Defense. Self-directed Violence (SDV)Classification System. 2010.Department of Veterans Affairs, Health Services Research and DevelopmentServices. Strategies for Suicide Prevention in Veterans. Washington DC:Department of Veterans Affairs; January 2009.Department of Veterans Affairs. Office of inspector general implementingVHA’s mental health strategic plan initiatives for suicide prevention. 2009.http://www.va.gov/oig/publications/reports-list.asp. Accessed July 29,2009.Hawks S, Smith T. Thomas H, et al. The forgotten dimensions in healtheducation research. Health Education Research 2008; 23:319-324.Jobes D. Managing Suicidal Risk: A Collaborative Approach. New York, NY:Guilford Press; 2006.Jobes D, Comtois K, Brenner L, Gutierrez P. Clinical Trial Feasibility Studies ofthe Collaborative Assessment and Management of Suicidality (CAMS). InR O’Connor, S Platt, J Gordon (eds), International Handbook of SuicidePrevention: Research, Policy & Practice. Chichester, UK, Wiley –Blackwell:2011.64
    65. 65. ReferencesMagruder K, York J, Jobes D, et al. Patient and provider outcomes of e-learning trainingin CAMS. EDU 08-424.Health Services R &D, Department of Veterans Affairs.8/1/09-7/31/12.Means B, Toyama Y, Murphy R, Bakia M, Jones K. Evaluation of evidence-basedpractices in online learning: A meta-analysis and review of online studies. U.S.Center for Technology in Learning, Office of Planning, Evaluation, and PolicyDevelopment, U.S. Department of Education 2009. Available at:http://www.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf.Accessed on January 12, 2012.Oordt M, Jobes D, Fonseca V, et al. Training mental health professionals to assess andmanage suicidal behavior: Can provider confidence and practice behaviors bealtered. Suicide and Life-Threatening Behavior 2009; 39:21-32.Report of the Blue Ribbon Work Group on Suicide Prevention in the VeteranPopulation. www.mentalhealth.va.gov/suicide_prevention/Blue_Ribbon_Report_FINAL_June30_08.pdf.Seal K, Bertenthal D, Miner C et al. Bringing the war back home: Mental healthdisorders among 103,788 US Veterans returning from Iraq and Afghanistan seen atDepartment of Veterans Affairs Facilities. Archives of Internal Medicine 2007;167:476-82.Sundararaman R, Panangala S, Lister S. Among Veterans- CRS Report to Congress Reportfor Congress. Washington, DC: Congressional Research Services, Domestic SocialPolicy Division; 2008.Valenstein M, Kim H, Ganoczy D et al. Higher-risk periods for suicide among VA patientsreceiving depression treatment: Prioritizing suicide prevention efforts. Journal ofAffective Disorders 2009; 112:50-58.Williams R, Gatien G, Haggerty B. Design element alternatives for stress-managementintervention websites. Nursing Outook 2011: 59: 286-291. 65
    66. 66. Contact InformationJan York, PhD, APRN, FAAN (CO-I)Nursing Researcher, UNM Visiting FacultyMUSC Research Professorjanet.york@va.govElizabeth Marshall, MD, MBAResearch Coordinator, Designerelizabeth.marshall3@va.govKathryn Magruder, PhD, MPH (PI)Research Health ScientistMUSC Professormagrudkm@musc.edu66