Participation Station (PS) is a peer-operated, drop-in mental health services center offering a variety of programming free of charge to the public. Their mission statement says they are there: To provide a program based environment that educates, empowers and enhances the lives of individuals moving toward recovery and mental wellness A funded program of the National Alliance for Mental Illness Lexington and is partially funded by Bluegrass Mental Health Mental Retardation Board Inc. Staffed by volunteers and part-time employees and short on resources as well as experience, they asked me to solve a problem.
That problem was actually several problems.They needed to diversify funding to promote sustainability and growth, as well as gain legitimacy in the community.“Prove” success after nearly 2 years of workConvince other agencies to collaborate on advocacy issues and the shared mission of recoveryWanted to begin pursuing internal CQI before a non-quality culture was established
Practices and places like Participation Station are relatively new. The variety of services offered by similarly minded organizations complicates attempts at traditional research. Oftentimes, researchers looking at similar venues would not be comparing apples to apples but would be comparing fruit salad to fruit salad. As a result, my literature review ended up focusing on the ideologies and ethics involved in the Recovery Movement and Consumer Movement – the foundations upon which PS was built. I was able to build a vocabulary and understand the technical jargon within the measurement tools which aided in my analysis.
Structured individual interviewInterviewer must be a “peer”Responses recorded by participantProgram Satisfaction Module is one of many that are offered as part of the protocol; choosing one module is acceptable and this module offered the most appropriate questions for what they are measuring. 1-4 for the first 40 questionsLast 10 questions included a coercion scale measure that had been included in the POP with permission from its creator
FACIT covers 6 domains of proficiency: Structure, Environment, Belief Systems, Peer Support, Education & AdvocacyPS was at or near benchmarks in 3:Environment, Belief Systems, Education and behind in 3 : Structure, Peer Support and Advocacy
Calculated mean of responses from PSCompared those to the benchmarked responses for each itemUsing the standard deviation from the benchmark data as a constant,Established a color-coded analysis.Items labeled “red”…Items labeled “yellow”…Items labeled “green”…Items labeled “purple”…
These deficiencies are universally related to the fact that Participation Station is a program of NAMI Lexington. They are technically overseen by the NAMI Lexington Board of Directors, meaning that their board is actually the NAMI Lex board – comprised of family members, practitioners, and other interested parties who are not “peers”. Hiring decisions & budget decisions are with NAMI Lexington’s management for liability reasons – they aren’t a separate entity and can’t have their own liability insurance, etc. so NAMI Lexington currently serves that role.
The line items here each reflect the newness of the program, its relationship with NAMI Lexington, and its uniqueness in the state of Kentucky. Training deficiencies account for 4 of the 5 items listed here. Consumer Staff is a reflection of being a NAMI programThere were no other consumer-operated service centers at the time of the FACITPersonal Empowerment & Recovery reflect a need for training of volunteers & staff in these areas. Informal Crisis Prevention is a call for standardization which also translates to training first and foremost.
Establishing excellence in some areas allows PS to shine on grant applications, influences program reviews by stakeholders, and expresses their benefit to the general community. In this instance, these are areas where internally the program’s leaders feel they are doing really well; they are areas to highlight, search for grants/awards in, and to continue tweaking so they can meet their goal of being a “flagship” peer support institution.
Team decided to keep track of individual responses so they could look at them in the future for individual outcomes measures as well.12 individuals took the POP before January 1, 2012. All were peers and had been coming to PS for at least 3 visits.
The overall results of the POP were positive. There were not many deviations from positive results on an individual basis.The Mean of all responses showed the employment issue.Median illustrated the fact that PS does not cover housing issuesIllustrates a clear lack of coercion on part of PS’s program leaders.
Recommendation: Engage an external evaluator as soon as you can!Continue performing FACIT on an annual or bi-annual basis to keep data current.
“Red” Recommendations Continue process of formally separating from NAMI Lexington, so peer leadership team is the top of org chartTrain leaders on more executive functions, so they are ready to take over “Yellow” RecommendationsTraining staff & volunteers in recovery principles, empowerment principles and engaging community in their missionEstablish a formal crisis prevention process“Purple” RecommendationsMonitor environment regularly to ensure non-coercionReach out to other community agenciesReach out to more traditional mental health practices
Structure a plan for referral of those needing employment assistanceHave information available on other services in the communityContinue to monitor measures that relate to the mission and values of Participation Station
1. Evidence-Based Recommendations for Participation Station Marcie Timmerman Capstone Presentation MHA Candidate, 2012 UK College of Public Health
2. Participation StationMISSION STATEMENT: To provide a program-basedenvironment that educates, empowers and enhancesthe lives of individuals moving toward recovery andmental wellness.
3. The Problem(s)• Need funding diversification: grant applications• Need to “prove” their success to date• Need to convince other agencies to collaborate for advocacy and mission• Pursuing internal continuous quality improvement
4. Project Purpose• Analysis of program evaluation and outcomes data collected Sept-Dec 2011• Recommendations for improvement based on that analysis
5. Literature Review• Gained understanding of models and ideologies which led to PS creation• Learned vocabulary and theories behind tools questions• Little to no available measurement tools, peer-reviewed “best practices”, etc. for services like PS in literature
6. Methods: FACIT• Fidelity Assessment Common Ingredients Tool (FACIT) from SAMHSA with benchmark data – Structured Observation – Structured item-by-item questionnaire Evaluations could be internal or external. • Internal only • Interviewed 12 leaders, 75% of whom were peers
7. Example of FACIT Survey Question1.1.3 “Hiring (1)Consumers are not involved in anyDecisions hiring decisions (2) Consumers have some involvement in hiring decisions (3) Consumers are responsible for making most of the hiring decisions (50% or more). (4) Consumers are responsible for making all hiring decisions.
8. Methods: POP• Peer Outcomes Protocol (POP) – Structured questionnaire with anchored scale responses for each participant • 1 for “Disagree” • 2 for “Somewhat Disagree” • 3 for “Somewhat Agree” • 4 for “Agree” – Last 10 questions “Coercion Scale” • Anchored 1-5 with “5” as best – Some questions worded so that the “4” is bad and “1” is good
9. Examples of POP Questions1. Overall, I am satisfied with this peer supportprogram.2. I am satisfied with the peer program facilities,such as the condition and layout of the rooms andbuilding.4. Overall, the program services are useful to me.
10. FACIT Results & Analysis
11. FACIT Results: Year One Comparison of Domain Scores to National Benchmark 50 45 Yr 1 Benchmark 40 35 30Total Score 25 20 15 10 5 0 Structure EnvironmentBelief Systems Peer Support Education Advocacy Domain
12. FACIT Analysis “Red” Analysis: 2+ below benchmark, Areas of deficiency “Yellow” Analysis: 1-1.99 below benchmark, Close, but not quite “Green” Analysis: Proficient“Purple” Analysis: 1 or more above benchmark, Excellence
13. "Red" Scored Data Year 1 5 Benchmark 4.5 4 3.5 3MEAN 2.5 2 1.5 1 0.5 0 1.1.1 Board 1.1.4 Budget Control 1.1.3 Hiring Decisions Participation ITEM
14. Red Analysis & RecommendationsSTRUCTURE DOMAIN:• Board Participation• Hiring Decisions• Budget Control
15. “Yellow” Scored Data 6 5 Year 1 Benchmark 4MEAN 3 2 1 0 1.3.2 Linkage to 3.3.1 Personal 1.1.2 Consumer Staff 3.5 Recovery 4.4.2 Informal Crisis Other Consumer- Empowerment Prevention Operated Services ITEM
16. Yellow Analysis & RecommendationsSTRUCTURE DOMAIN• Consumer Staff• Linkages with other Consumer-Operated SrvcsBELIEF SYSTEMS DOMAIN• Personal Empowerment• RecoveryPEER SUPPORT DOMAIN• Informal Crisis Prevention
17. “Purple” Scored Data 6 Year 1 5 Benchmark 4MEAN 3 2 1 0 3.4 Choice 3.7 Spiritual Growth 1.3.1 Linkage with TMHS of Coerciveness Other Service Agen 2.2.1 Lack 1.3.3 Linkage with ITEM
18. Purple Analysis & RecommendationsSTRUCTURE DOMAIN• Linkage with Other Service Agencies• Linkage with Traditional Mental Health ServicesENVIRONMENT DOMAIN• Lack of coercivenessBELIEF SYSTEMS• Choice• Spiritual GrowthEDUCATION DOMAIN• Formal curriculums in problem-solving and self- management
19. POPResults & Analysis
20. POP Results• Mean: Goals met in all categories except meeting employment needs of participants• Median: “Staff threatens me with loss of housing” is irrelevant to services at PS, had a median indicating “not applicable”• Mode: Coercion scale uniformly positive with “5” as mode – best score possible.
21. Project LimitationsFACIT• Internal review only• Performed over several weeks• No internal comparison (1st time)POP• Interviewer error• Small sample size• No benchmark data available
22. Recommendations: FACIT• “Red” Recommendations – Continue separation plan – Training• “Yellow” Recommendations – Training – Formal crisis prevention process• “Purple” Recommendations – Monitor environment – Reach out
24. Implementation Plan• Establish dedicated quality improvement team from peers• Develop key indicators to monitor which are relevant to mission and applicable to funding requests• Small cycles of change – path of least resistance to start• Additional training on CQI• Inform stakeholders of results
25. Benefits to Participation Station• Usable data for grant applications• Evidence-based practices• Training needs outlined• Small cycles of change start with the recommendations