The Psychometric Qualities of a Consumer Quality Index Assessing Clients’ Experiences with Services for Homeless People, Homeless Youth and Battered Women
Presentation given by Marielle Beijersbergen, Jolanda Asmoredjo & Judith Wolf, The Netherlands at a FEANTSA Research Conference on "Understanding Homelessness and Housing Exclusion in the New European Context", Budapest, Hungary, 2010
The Psychometric Qualities of a Consumer Quality Index Assessing Clients’ Experiences with Services for Homeless People, Homeless Youth and Battered Women
1. Consumer Quality Index for Shelter
Services
Measuring client experiences with shelter services
for homeless adults, homeless youth and battered women
in the Netherlands
Judith Wolf
Mariëlle Beijersbergen
Jolanda Asmoredjo
Milou Christians
Research Centre for Social Care (Omz)
3. 1. Background
What is a Consumer Quality Index?
• Standardised method to measure client experiences with health
care.
• Strict guidelines for development, use of instrument, data
analyses and reporting (developed and monitored by Centre for
Client Experience with Care - CKZ).
• Two types of CQ-questionnaires: experience and importance
• Measurements can be used for internal quality control and
external accountability purposes.
• Information for clients, shelter organisations, government, etc.
4. 1. Background
Importance of a CQI for shelter organisations
• Little known about how clients experience shelter services
• Little known about what clients find important about care
• Sector uses variety of instruments:
• developed for other target groups
• no standardised instruments (reliability, validity, etc)
• no comparisons possible
Necessities:
• Standardised instrument
• Possibility for benchmarking
5. 1. Background
Preparatory research pilot CQI Shelter services
Sources:
• Literature search
• Expert interviews
• Focus group discussions with stakeholders (clients, workers)
• Concept mapping method (clients, workers)
Process:
• Pre-pilot & pilot survey
• Monitored by Centre of Client experiences with Care (CKZ) &
advisory council
6. 2. Pilot survey
• 3 target groups: battered women, homeless adults and
homeless youth (20 shelter facilities per target group)
• 2 versions of CQI-SS: ambulant (67) & residential (78)
• Pilot questionnaire: 42-52 items on clients’ experience:
- Contact with worker
- Appropriate care & counselling
- Information
- Children
- Received care
- Outcomes of care
- Living conditions (residential version)
• Standard CQI answer categories: 1=never - 4=always
7. 2. Pilot survey
Response pilot questionnaire
• 848 744 questionnaires to be used (+ 116 retest)
• Battered women 34%
• Homeless adults 37%
• Homeless youth 29%
• 12 respondents per shelter
8. 3. Psychometric analyses
Factory analyses
• 3-4 reliable scales (Cronbach alphas .72 - .91)
Table: Cronbach’s alphas per CQI scale
Hom. Bat. Hom.
Scale (nr. items) Total adults women youth
(N) (N) (N) (N)
Client-worker .84 .85 .81 .86
interaction (4) (693) (269) (246) (215)
Received care .90 .90 .89 .91
(10) (693) (255) (231) (207)
Outcomes of .87 .87 .87 .86
care (5) (709) (264) (237) (208)
Living conditions .77 .79 .72 .77
(residential) (6) (499) (207) (183) (109)
9. 3. Psychometric analyses
Retest stability
Table : Pearson’s Correlations Test-retest: sufficiently reliable over time
Scale Total Homeless Battered
(NRange=114-115) adults women
(NRange=57-58) (NRange=57-58)
Client interaction .67 .62 .71
Received care .67 .69 .66
Outcomes .52 .59 .47
Living conditions .62 .54 .69
(residential)
10. 4. Multilevel analyses
• Controlled for client background characteristics influencing CQI score
Table: Intraclass correlations (ICC’s)
Scale N clients ICC ICC
(units) controlled*
Client-worker interaction
Homeless adults 266 (18) .23** .17**
Battered women 231 (24) .03 .03
Homeless youth 216 (18) .15** .16**
Received care
Homeless adults 266 (18) .37** .31**
Battered women 231 (24) 0 0
Homeless youth 216 (18) .23** .22**
Results of care
Homeless adults 266 (18) .20** .15**
Battered women 231 (24) .05 .02
Homeless youth 216 (18) .07* .06
Living conditions (residential)
Homeless adults 213 (14) .35** .20**
Battered women 193 (20) .11* .11*
Homeless youth 117 (10) .16* .15*
11. 5. Adjusted instrument
Content CQI-SS
• Background questions
• (Living conditions in shelter)
• Client-worker interaction
• Received care
• Results of care
• Overall opinion of facility (grade 0-10)
• Final questions
12. 5. Adjusted instrument
Table : CQI-SS: Scales & items measuring cliënt experiences
Scale Number Example
of items
Client-worker 4 Does the employee take you seriously?
interaction
Received care 10 Do you receive as much assistance as you
need?
Results of care 5 As a result of the assistance, are you able to
make better decisions about your life?
Living conditions 6 Is it clean in the facility?
(residential)
Separate items 13 - Do you receive advice about your safety?
- Have any agreements been made with you
about the supervision of your children?
- How do you rate this facility? 0 means: really
poor. 10 means: excellent
13. 6. Future developments
CQI-Shelter Services from 2011
• CQI-SS will be publicly available in 2011, on website
CKZ
• CKZ will provide Quality mark of CQI if
measurements are done with approved CQI-instrument
by certified research organisations.
• Building up benchmark dataset: comparing within and
between organisations (over time)
14. Conclusion
• CQI-SS measures client experiences from the perspectives
of the clients.
• Widely usable instrument for homeless adults, homeless
youth and battered women
• Good psychometric properties: reliable and with high
discriminative power (and valid – results not presented)
15. Thank you for your attention
Contact
Judith Wolf
j.wolf@elg.umcn.nl
Editor's Notes
Foto’s invoegen
Niet tevredenheid.. Niet iedereen hoeft het wiel opnieuw uit te vinden Wetenschappelijk onderbouwd, met nadrukkelijke inbreng van het veld opgesteld Vergelijkbare informatie over verschillende zorgaanbieders Kwaliteitseisen
Niet tevredenheid.. Niet iedereen hoeft het wiel opnieuw uit te vinden Wetenschappelijk onderbouwd, met nadrukkelijke inbreng van het veld opgesteld Vergelijkbare informatie over verschillende zorgaanbieders Kwaliteitseisen
Niet tevredenheid.. Niet iedereen hoeft het wiel opnieuw uit te vinden Wetenschappelijk onderbouwd, met nadrukkelijke inbreng van het veld opgesteld Vergelijkbare informatie over verschillende zorgaanbieders Kwaliteitseisen