Using Multiple Technology Platforms to Collect Data for the Evaluation of an mHealth Intervention to Improve Community Health Nurse Motivation in Ghana
•
1 like•159 views
Presentation on using multiple platforms to collect data for the evaluation of an mHealth intervention to improve community health nurse motivation in Ghana made at the 2017 American Evaluation Association conference.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Using Multiple Technology Platforms to Collect Data for the Evaluation of an mHealth Intervention to Improve Community Health Nurse Motivation in Ghana
Similar to Using Multiple Technology Platforms to Collect Data for the Evaluation of an mHealth Intervention to Improve Community Health Nurse Motivation in Ghana (20)
More from JSI
More from JSI (20)
Recently uploaded
Recently uploaded (20)
Using Multiple Technology Platforms to Collect Data for the Evaluation of an mHealth Intervention to Improve Community Health Nurse Motivation in Ghana
- 1. Sophia Magalona Soumya Alva JSI Research & Training Institute November 9, 2017 American Evaluation Association , Washington DC Using multiple technology platforms to collect data for the evaluation of an mHealth intervention to improve community health nurse motivation in Ghana
- 2. 2| Increasing need for real-time, high quality data for effective monitoring and evaluation in global health programs Data collection activities are often perceived as time-consuming and complex, and limited by resource constraints. Data are frequently fragmented, inaccessible, incomplete, and prone to error. Introduction
- 3. 3| CHNs - frontline health workers of the Ghana Health Service • Face many challenges • Work in rural, underserved areas • Affects their motivation and morale • Affects retention and quality of care CARE COMMUNITY HUB (CCH): an innovative solution to address barriers in health worker job satisfaction and motivation through the use of ICT via a smartphone application (CHN on the Go). It seeks to improve connectivity and career development among CHNs, increase their linkage and interactions with a professional network and supervisors, and provide clinical refreshers Designed using a Human Centered Design approach The evaluation was an iterative learning process Background
- 4. 4| Implemented by Grameen Foundation 5 DISTRICTS: Ningo Prampram, Ada East and West (Greater Accra), and South Tongu, South Dayi (Volta) Approximately 50 health centers and associated CHPS More than 300 CHNs and supervisors Project timeline: • Design process: Oct 2013-March 2014 • Version 1 June 2014 • Version 5 May 2016 GHANA VOLTA Togo Burkina Faso Côte d’Ivoire GREATER ACCRA Project Description
- 5. 5| CCH Phone and CHN on the Go App
- 6. RMEApproach
- 7. 7| Learning Center Health knowledge Point of Care Health knowledge, relationships Planner Targets, Work scheduling Achievement Center Targets, relationships Staying Well Keeping well, relationships WHATSAPP Communication & Connectedness Supervisor App Monitoring & Supervision AnAdaptationoftheTheoryofChange Driver (Direct outcomes (Modules & Intermediate outcomes) Direct outcomes ) Driver Change Pathways FINAL OUTCOMES: Better CHN performance Better quality of care Improved maternal & child health ASSUMPTION: Health workers feel unmotivated and frustrated in their jobs SYSTEM LEVEL CHANGES: Supervision, education/learning
- 8. 8| What are baseline and endline levels of CHN motivation and job satisfaction in rural Ghana? Is the use of an mHealth platform such as CHN on the Go associated with greater health worker motivation/satisfaction? Was the project implemented as planned? Barriers and supportive factors that affected implementation? Expected results as foreseen in the pathways in the TOC? 1 2 3 ResearchQuestions
- 9. 9| Data Baseline Endline Dates of data collection May – August 2014 March – April 2016 In-depth interviews 29 CHNs 11 Supervisors 40 CHNs 8 Supervisors 12 Stakeholders 2 Clients Focus group discussions with CHNs 4 (23 CHNs total) Job Satisfaction & Motivation Questionnaire 186 CHNs 185 CHNs Knowledge Questionnaire 184 CHNs 186 CHNs Data Sources Magpi e-survey Usage Dashboard (Tableau) Pop-up questions Process Documentation Feedback sessions Monthly Progress Reports
- 10. 10| RME Timeline Project Start May – June 2014 Project End May 2016 Baseline May – Aug 2014 PD Round 1 May – Aug 2014 Baseline May – Aug 2014 Magpi e-survey April – May 2015 Pop-up Questions June 2015 PD Round 2 July – August 2015 Magpi e-survey Dec 2015 – Jan 2016 Endline April 2016 Pop-up Questions May 2016 Other data collection activities: • Monthly field interviews • Monthly usage monitoring dashboard
- 11. 11| Baseline Endline CHNs (n) 186 186 CHNs trained and provided with devices 264 Districts (%) Ada East/West 29 25 Ningo Prampram 24 23 South Tongu 25 28 South Dayi 23 23 Age (%) 18-24 years 10 7 25-30 years 68 63 31-39 years 15 25 40-49 years 3 2 50+ years 4 3 Sex (F) (%) 92 97 Works at CHPS (%) 52 50 Married (%) 38 45 Supervisors trained: 62 CHN Background
- 12. Summary Findings
- 13. 13| 0 50 100 150 200 J 2015 F M A M J J A S O N D J F M AM 2016 CCH Phone Usage High Medium Low Non-use Number of CHNs
- 14. 14| Designed with user input in mind CCH Phone and App usage • CHNs valued CCH & CHN on the Go app, despite some difficulties • Supervisors recognized the benefit of the app • Point of Care and Learning Center more popular than other modules • WhatsApp was a particularly useful mechanism for communication Changes in overall satisfaction levels among CHNs not as high as expected • 4% increase in satisfaction over time • 5% increase in satisfaction over time in younger (<30 years old) CHNs • 94% said the app met their needs • Association of job satisfaction with app use not always clear Summary Findings
- 15. 15| App addresses various needs of CHNs & supervisors As a job aide for CHNs (work planning, knowledge, communication) As a job aide for supervisors (communication, monitoring & supervision) Helps CHNs • Gain confidence • Gain respect & recognition from colleagues, supervisors & clients • Provides emotional support for some Summary Findings Sustainability • CHNs want to continue using the app - both current and other districts • GHS supports scale up, using CHNs own phones, especially in Volta region
- 16. 16| Phones freezing and the CHN on the Go app crashing Batteries run out quickly: almost half (47%) of respondents said that their phones go off 1-2 days a week because of not charging Other reported problems included small screens that make it difficult to see content CHNs not having enough credit to use their phones Summary Findings
- 17. 17| Complex app with many modules Collation of all data was sometimes challenging No benchmarks/standards set against which to measure success Network connectivity problems Challenges
- 18. 18| Purposefully and successfully addresses the needs of users Valued by CHNs, willingness to use own funds for future use Trained and equipped with devices • Over 300 CHNs and their supervisors • National (Family Health, HR, & PPME), regional/district (health mgmt. teams) Close collaboration with GHS (FHD, PPME, HR, ICT) Pioneered e-learning in Ghana • Nursing and Midwifery Council accreditation with GHS approval for continuing professional education Successes
- 19. Thank you!
Editor's Notes
- Increasing need for real-time, high quality data for effective monitoring and evaluation in global health programs in order to improve program management, track progress, determine program effectives and scale-up successes; however, data collection activities face several challenges, often perceived as time-consuming and complex, and limited by resource constraints. As a result, data are frequently fragmented, inaccessible, incomplete, and prone to error. this can be challenging in resource poor settings where existing data sources are often incomplete or outdated This paper highlights the use of multiple technology platforms to provide close to real time, high quality, accessible and actionable information throughout the implementation of an mhealth intervention, providing an opportunity to understand the pathways through which the outcomes are reached in order to make programmatic changes through the life of the project.
- who are paid frontline health workers posted at CHPS compounds or health facilities and provide community-based preventive and curative maternal, newborn, and child health care while residing in the community. CHNs face many challenges in their work, including isolation in remote areas, limited access to important health information and professional advancement and learning opportunities, all of which affect the motivation of CHNs, ultimately impacting retention rates and quality of care. The Care Community Hub (CCH) pilot was part of Concern Worldwide’s Innovations for Maternal, Newborn & Child Health initiative funded by the Bill and Melinda Gates Foundation in five districts in Ghana. The pilot aimed to provide the government of Ghana with an innovative …………… ----The evaluation of the CCH project was an iterative learning process, with the need for data to be regularly monitored and subjected to periodic discussions that eventually feed into revising project design. Community-based Health Planning and Services (CHPS) CHPS compounds are Ghana’s primary strategy to extend health care provision to those who have been beyond the reach of the existing formal system. Community health nurses (CHNs),
- Modules designed to facilitate health worker learning, supervision and performance monitoring, communication and work activity planning, as well as to support health worker wellness.
- The TOC guided the evaluation of the CCH project, and over time results from periodic data collection showed the complex relationship between the CCH app modules and the components of motivation. To simplify this relationship, the research team created a diagram (see below) to illustrate the change pathways through which the CCH app could affect health worker motivation.
- Mainly sources of qualitative data – wanted to supplement this with close to real time data from 3 platforms. These platforms include a Magpi survey administered via mobile phones, which collected information on self-reported app usage and operational indicators including technical issues pop-up questions, which were short questionnaires sent directly to nurses’ phones for self-response on satisfaction with the app and motivation; and a Tableau dashboard, which visualized overall usage of the app and also of different modules of the app per month. Regular qual and quant data were triangulated, summarized, and fed back to program staff to inform implementation and provide an evidence-base for ongoing improvements.
- Illustration of our RME timeline – to show the sequence of data collection activities
- Quick snapshot of the CHNs in the project. Overall 264 were trained and provided phones, due to movement/turnover, there were 186 CHNs present during baseline and endline. Majority were between 25-30 years and female Approximately half were living in CHPS compounds
- Actual usage of the CHN on the Go App over time fluctuated, with the highest use in early 2015 Opening the app for 1-5 days per month (low), 6-10 days per month (medium) and at least 11 days per month (high) Usage may have been influenced by the fact that both supervisors and nurses were not given any specific guidance or requirements on the extent to which they needed to use the app. Their use of the app and the modules was defined by their sense of how useful it would be for their work needs. As a result, usage was possibly low in certain months because of other work demands, such as the preparation of annual reports in the months of October to January, when a greater number of non-users was observed. Overall, in all months, there were nurses who did not use the app.
- CHNs had reported some problems with their phone that affected their use. There were problems with phones freezing and the CHN on the Go app crashing; however, new versions have been improved. CHNs also pointed out that batteries run out quickly, and the Magpi e-survey showed that almost half (47%) of respondents said that their phones go off 1-2 days a week because of not charging. Other reported problems included small screens that make it difficult to see content and CHNs not having enough credit to use their phones.
- Developing the app also turned out to be much more complex and time-intensive than initially planned. The first version of the app was quite basic in comparison with the final version of the app. With five versions of the app, modifications to the app were made at regular intervals based on user input and needs of district-level GHS staff right up to the last few months of program implementation. Made it challenging to evaluate While the availability of data from multiple sources was very useful to present a more complete picture, the collation of all these data was sometimes challenging, especially when they did not always provide a consistent story. For example, the available monitoring data did not always match the findings from qualitative data. No benchmarks/standards were set for the optimal/minimal level of use of the CCH phone and app expected of users. CHNs used the app based on their need. For example, there was no requirement for CHNs to complete a set number of courses every month. Also, while some CHNs updated their Planner with activities every week, some did it less frequently. Although CHNs may have updated the Planner only once or twice a month, they received automatic reminders from the phone which helped them perform their job. However, these reminders from the phone did not get logged in as usage of the app. As a result, it was very challenging for the project to measure success. Possible that for some users, usage data had problems syncing with server
- By the end of the project, more than 300 CHNs and their supervisors had been trained to use the app. Training was also provided to staff at the national level (Ghana Health Service [GHS] Family Health Division, Human Resources and Policy Planning Monitoring and Evaluation [PPME] departments), as well as the regional/district health management teams. The Grameen Foundation and Innovations for MNCH teams in Ghana made considerable effort to obtain buy-in at all levels of the GHS. Stakeholders at the national, regional, and district levels were engaged and are very supportive of the CHN on the Go app and its use going forward. The teams made considerable effort to link with the education system for nurses, health schools such as the University of Health and Allied Sciences for distance learning and the Nursing and Midwifery Council to provide accreditation for courses by CHNs. GHS sees a lot of value in the CHN on the Go app, and the government is committed to absorb future server costs to ensure sustainability of the app in all districts.
- In terms of motivation, the app was NOT designed to address issues and concerns of CHNs related to logistics, transportation, salary, etc., which are systemic factors that make their work challenging and negatively influence their motivation to perform