Shedding Light on the Reliability and Validity of PRISM Tools Anwer Aqil , MD, MCPS, MPH, DrPH Senior HIS Advisor MEASURE Evaluation

Anwer Aqil , MD, MCPS, MPH, DrPH

Senior HIS Advisor

MEASURE Evaluation

Objectives Assess whether PRISM tools produce consistent and valid results Provide comparisons of PRISM tool findings in 2004 and 2007 to assess consistency Share PRISM lessons learned

Assess whether PRISM tools produce consistent and valid results

Provide comparisons of PRISM tool findings in 2004 and 2007 to assess consistency

Share PRISM lessons learned

Technical Factors Complexity of the reporting form, procedures HIS design Computer software IT complexity Behavioral Factors Data demand Data quality checking skill Problem solving for HIS tasks Competence in HIS tasks Confidence levels for HIS Tasks Motivation Organizational Factors Governance Planning Availability of resources Training Supervision Finances Information distribution Promotion of culture of information HMIS Processes Data collection Data trans- mission Data Processing Data Analysis Data presentation Data quality checking Feedback Improved Health System Performance Improved health status Improved HMIS Performance Data Quality Information Use Performance of Routine Information System Management (PRISM) Framework PROCESSES OUTPUTS OUTCOME INPUT IMPACT

Technical Factors

Complexity of the reporting form, procedures

HIS design

Computer software

IT complexity

Behavioral Factors

Data demand

Data quality checking skill

Problem solving for HIS tasks

Competence in HIS tasks

Confidence levels for HIS Tasks

Motivation

Organizational Factors

Governance

Planning

Availability of resources

Training

Supervision

Finances

Information distribution

Promotion of culture of information

HMIS Processes

Data collection

Data trans- mission

Data Processing

Data Analysis

Data presentation

Data quality checking

Feedback

Improved HMIS Performance

Data Quality

Information Use

Methodology First survey was conducted in 12 districts and 110 facilities in 2004, using LQAS and repeated in 2007 Triangulation of data from different sources and levels to improve validity Face and content validity is measured by expert review of tools Checking data quality and information use through record review and observation is considered a gold standard for validity Reliability (consistency) of observational tools is assessed by comparing results in 2004 and 2007 Construct validity is assessed by testing relationships among different PRISM framework constructs Reliability of constructs scales such as confidence level, promotion of culture of information is measured by internal consistency or Cronbach alpha

First survey was conducted in 12 districts and 110 facilities in 2004, using LQAS and repeated in 2007

Triangulation of data from different sources and levels to improve validity

Face and content validity is measured by expert review of tools

Checking data quality and information use through record review and observation is considered a gold standard for validity

Reliability (consistency) of observational tools is assessed by comparing results in 2004 and 2007

Construct validity is assessed by testing relationships among different PRISM framework constructs

Reliability of constructs scales such as confidence level, promotion of culture of information is measured by internal consistency or Cronbach alpha

HMIS Performance: Data Quality and Use NA NA 15 - Facilities having documents to show use of HMIS data 54 - Facilities showing some decision-making based on HMIS data 38.2 54.2 Display of HMIS data for monitoring 2007 2004 Use of information (N=110)

Behavioral Determinants Other Behavioral Determinants- Average motivation levels for HMIS tasks were 41.2% and 75% in 2004 and 2007, respectively Average HMIS tasks confidence levels – 65-80% in 2004 and 56-65% in 2007

Other Behavioral Determinants-

Average motivation levels for HMIS tasks were 41.2% and 75% in 2004 and 2007, respectively

Average HMIS tasks confidence levels – 65-80% in 2004 and 56-65% in 2007

r=.16, p=.000 - Association between culture of information and HMIS task competence r=0.32, p=.000 r=0.36, p=.000 Association between perceived promotion of culture of information and perceived HMIS tasks confidence Validity - Concurrent 0.85 0.87 Overall perceived culture of information 0.86 0.95 Overall perceived HMIS tasks confidence (self-efficacy) 2007 2004 Reliability - Cronbach Alpha Reliability and Validity of Perceived HMIS Tasks Confidence & Promotion of Culture of Information Scales

Lessons Learned (1) PRISM tools provide a holistic picture of the HMIS, providing information not only on HMIS performance but also its behavioral, technical and organizational determinants PRISM tools - adaptable and applicable to country situation; in any types of information system – community, facility, hospital, etc. When to use PRISM tools - For HMIS reform, baseline assessment or evaluation, use all PRISM tools For monitoring HMIS performance and processes over time, apply diagnostic tool For monitoring HMIS functions, use MAT For monitoring behavioral and organizational factors, employ OBAT

PRISM tools provide a holistic picture of the HMIS, providing information not only on HMIS performance but also its behavioral, technical and organizational determinants

PRISM tools - adaptable and applicable to country situation; in any types of information system – community, facility, hospital, etc.

When to use PRISM tools -

For HMIS reform, baseline assessment or evaluation, use all PRISM tools

For monitoring HMIS performance and processes over time, apply diagnostic tool

For monitoring HMIS functions, use MAT

For monitoring behavioral and organizational factors, employ OBAT

Lessons Learned (2) Sampling – Use a sample size of 19* to assess whether standards are achieved based on LQAS Use a sample size of 100 to calculate HMIS performance level and associated determinants based on LQAS 30 Cluster sampling Convenience - when costs and time are limited; generalization of results limited Costs - based on the purpose and scope Monitoring – make tool part of the existing supervisory system Training – 3 day TOT on tools for 20 persons, US$5000-10,000 Time for baseline/evaluation – total 4-6 weeks, (preparation, fieldwork and writing report)

Sampling –

Use a sample size of 19* to assess whether standards are achieved based on LQAS

Use a sample size of 100 to calculate HMIS performance level and associated determinants based on LQAS

30 Cluster sampling

Convenience - when costs and time are limited; generalization of results limited

Costs - based on the purpose and scope

Monitoring – make tool part of the existing supervisory system

Training – 3 day TOT on tools for 20 persons, US$5000-10,000

Time for baseline/evaluation – total 4-6 weeks, (preparation, fieldwork and writing report)