Project contribution to AFP surveillance can be difficult to measure – in this evaluation, not measured directly, but by knowledge and attitudes. What is the opinion of key informants about the work of the CORE Group Polio Project?
All conducted by consultants. Limitations – no comparison group. Two reasons – difficult to find a comparison group – work in high-risk areas; chosen in consultation with WHO/MOH. Cost.
***Mention that we are not comparing countries ***All data from household surveys
Our first slides look at whether caregivers recall having contact with a CGPP volunteer/worker. And we decided to include two of our primary social mobilization activities: Door-to-door counseling and group education sessions. So, the first graph here represents the question do you recall being visited by a CGPP volunteer/worker at times other than during a vaccination campaign? Figures in orange are for Angola. We actually collected this data at baseline in Angola, and the numbers seem to be comparable. Dark blue is the midterm figure for Ethiopia and the dark green is the midterm figure for India. [Read numbers]. We followed this question by asking those that recalled being visited what they recalled discussing. You can see that the vast majority recalled discussing a polio-related topic. And in fact, in Angola, this figure rose significantly between baseline and midterm.
This is identical to the previous slide except that we’re looking at figures for group education sessions. At baseline, we collected this data in India, and so you’ll see we have comparative data for India. You’ll see that these are a bit lower than door-to-door counseling. But again, the vast majority recall discussing a polio-related theme. So, together these figures could suggest that while we don’t necessary have the reach that we would like with these activities, the contact that does occur is of such quality that caregivers would spontaneously recall discussing polio-related topics.
One of our attitude questions on the survey. Obviously, this is one question where we would like to see percentage drop. Angola started at the highest of the 3 countries and dropped more than 10 percentage points in 2 years. (statistically significant). Largest percentage of caretakers said that children with disabilities should not be vaccinated. Followed by some confusion on age at which should be vaccinated. Presumably coming from campaign age cut-off. Make sure caretakers know that it’s not harmful older than age 5 – actually campaigns in central Africa have been targeted to older children and adults because of age affected. Ethiopia began at low levels and essentially remained steady, within about one percentage point. India rose a couple of percentage points. Disappointed to not see this number drop in India. However, we ask a follow on to this question (which children). We found that those who said newborns dropped from 10% of all caretakers surveyed to 0%. Unvaccinated newborns have been identified as a major barrier to eradication in India and this is key message we’ve adopted. Almost all respondents (98%) said “sick children” , so we’ll be working on this message.
Reiterate what community volunteers/workers do to support routine immunization??? The next few slides look at vaccination rates between baseline and midterm. Here we’ve calculated vaccination rates based on what has been recorded in a child’s vaccination card. So, these figures are conservative estimates of vaccination rates because there will be children who have been vaccinated, but don’t have it recorded or don’t have a card. We have some baseline/midterm comparative data on vaccinations rates by card and caretakers recall, although the data are not as complete. The absolute figures differ, but the trends that we see are largely the same between baseline and midterm. So, what I’ll do is for each of the three countries, first show you the card retention rate so you have a sense of the numbers in each country who had vaccination cards for their children. And then we’ll look at the vaccination rates based on these cards.
Angola – caretakers produced vaccination cards for about half of all children in the survey.
Here you can see vaccination rates for 7 vaccinations (OPV0 – OPV3 and Penta1 – Penta3). The light orange are baseline figures and the dark orange are midterm figures. You can see an overall positive trend in vaccination rates across all the vaccinations. You’ll note statistically significant differences between baseline and midterm for both OPV3 and Penta3. This is interesting and very positive actually because it suggests that there are fewer drop-outs. For example, at baseline there was a 15 percentage point drop between Penta1 and Penta3. At midterm the difference was only 7 percentage points.
Let me start by mentioning that in Ethiopia we were able to sample enough caretakers so that we could look at estimates for 3 different general regions in Ethiopia. Our Ethiopia Secretariat team thought it most beneficial to break up the project area by predominant lifestyle because this influences how our volunteers work with their communities and health posts. So, if there were large and interesting differences between the three areas, I’ve broken them out. If not, then I used a weighted aggregate estimate. You can see here card retention rates for the three areas.
You can see here again the three regions (agrarian, semi-pastoralist, and pastoralist). To keep it simple, we’re showing you four vaccinations for each (OPV0 in blue, OPV3 in green, Penta1 in orange, and Penta3 in yellow). Other OPVs and Pentas follow the same pattern and that is, project areas saw a statistically significant increase in OPV0 rates in all three areas, which is great news in Ethiopia where, as you can see, OPV0 rates are much, much less than other doses because of cultural and logistical factors. Beyond, OPV0, agrarian areas saw small decreases or small increases in vaccination rates, but nothing significant. Whereas the two other areas, saw statistically significant increases in vaccination rates for all doses of OPV and Penta. Catching up to agrarian areas, which remained steady. All OPV0 significant; OPV3 S and P significant; Penta1 S and P significant; Penta3 S and P significant. BCG and measles – data quality a question for baseline. Same pattern OPV1 – Penta3 – S and P significant while A is not.
And finally, we have India. India’s card retention rate remained steady at about 30% of all children.
And here we have vaccination rates for project areas in India. The light green represents baseline figures and dark green represents midterm figures. You can see that the overall trend is positive, although, in some cases only by tenths of a percentage point. Measles at the very end is the only vaccination with a statistically significant improvement over the two years. You could question whether something similar to Angola is happening at a smaller magnitude, and that is that perhaps the data are suggesting that children are finishing the vaccination series more than they had been. The latter vaccinations (OPV3, DPT3, and Measles) had the largest, albeit still small gains, when compared to other vaccinations. Measles is only significant.
Angola – mention registries and how they were being implemented prior to midterm, but not fully operational in all project sites until just after midterm. Hope to see improvements from this initiative in the future. Ethiopia – Began integrating specific messages about OPV0 into work shortly after midterm. Ethiopia – Ethiopia has less formal registry system than other two countries. Can be successful in areas where health post well-organized. Implementing more formal system in other areas can help build capacity of health post. Problem is low and non literacy of volunteers in Ethiopia. India – Community mobilization coordinators stay busy with near monthly campaigns. Could there not be time for other activities? Are they overstretched? India – Another avenue to explore – whether accepting messages that CMCs give, but other barriers that prevent action. Example – Drop in those who thought that newborns shouldn’t be vaccinated, but did not materialize in OPV0 vaccination rates.
Onto our next leg of polio eradication – supplementary immunization activities. First thing to note here is that we have not included Ethiopia data here. There are very few campaigns in Ethiopia as their last case was reported 3 years ago. The comparisons between baseline (conducted just after last cases) and midterm is not meaningful. We have three indicators to explore supplementary immunization activities. First, caretakers were asked whether their child ever received polio vaccination in a campaign. Second, they were asked whether their child was vaccinated in the last campaign. And third, they were asked whether a vaccinator visited their home during the last campaign. Baseline figures are in the lighter colors (light orange for Angola and light green for India). Midterm figures are in darker colors (dark orange for Angola and dark green for India). So let’s begin with Angola in the orange. Campaigns and campaign quality were identified as the largest barrier and therefore the biggest priority in Angola to stop the ongoing transmission of polio there. It’s an ongoing struggle in Angola where political will has been weak. Dora can attest to this. You can see by the numbers that there have been significant improvements in all three SIA indicators in Angola in the two years since baseline. [Go over figures for first two indicators]. As a project, we do not have as much control over the third indicator. CGPP volunteers are not the vaccinators in either Angola or India. Our NGO staff helps to coordinate and our volunteers often accompany vaccination teams. In India, supplementary immunization activities happen almost monthly in our project areas. It’s a finely coreographed dance in which CMCs play an integral part. Through various activities, they mobilize community to participate before and during campaign; they work polio booths during booth days; and then work as part of vaccination teams, in particular helping to gain entry into resistant houses. India must have high campaign coverage in order to eradicate polio; and so you can see from the numbers that in project areas it’s really about maintaining high participation levels. [Go over numbers for first two indicators]. Finally, we, and in particular the India team, were shocked to see the figures for the final indicator. 100% of households should be visited by vaccinators during campaigns. The figure of 80% does not fit with internal project figures or official campaign figures external to the project. And we can only explain it by thinking it’s just an artifact in the data.
And on to our final polio eradication leg and our final three indicators from the household survey. Caregivers were asked whether they had (1) ever heard of Acute Flaccid Paralysis or sudden paralysis in children. Just a little background, not all cases of AFP are polio cases, but we use the AFP case definition to find children and test them further for polio, which includes collecting a stool sample in a timely way and sending it to a certified laboratory. If the caretaker had ever heard of AFP, we also asked them to describe the signs of AFP, which are essentially limp limbs and/or the child suddenly stops walking or crawling. And finally we asked those who had heard of AFP who they would contact is they suspected that their child had AFP. Here we’re wanting them to say that they would contact a health facility (formal health system) or a CGPP project volunteer (rather than, for example, a traditional healer, in which case the case may never remain invisible to the formal surveillance network. The graph you see looks at figures for those three indicators for both Angola and India (figures for Ethiopia are on the next slide). Figures for Angola are again in orange (light is baseline; dark is midterm) and figures for India are in green (again light for baseline and dark for midterm). Angola – You’ll see that the percentage who had ever heard of AFP dropped about 9 percentage points in the past two years, which is a statistically significant drop. Those who would contact a health facility or project volunteer also dropped – about 5 percentage points. India – Figures for India remained steady across all three indicators, with only tenths of a percentage point between baseline and midterm.
Finally, we have the figures for the same three indicators for the three regions of Ethiopia. Figures for Agrarian areas are in orange. Figures for Pastoralist areas are in blue. And figures for semi-pastoralist areas are in green. Please note that we have not done statistical testing for the second two indicators – you can imagine that some of the differences between baseline and midterm are statistically significant – we just haven’t yet run the numbers for these two indicators for Ethiopia. By and large these figures are very disappointing. Pastoralist areas have the least concerning figures. They’ve either dropped slightly as in the case of heard of AFP and signs of AFP or risen slightly in the case of AFP contact. You’ll find roughly 10 to 13 percentage point drops in all three indicators for agrarian areas. And you’ll see the largest drops in semi-pastoralist areas, which are very concerning. When we spoke with our Secretariat team in Ethiopia about these figures, their thought was that complacency accounts for much of the drop. As I’ve mentioned, Ethiopia has not seen a case of wild poliovirus since the spring of 2008. And the project in Ethiopia has focused more in the past two years on improving routine immunization. Interestingly, the 3 cases which occurred just before baseline, occurred in semi-pastoralist areas where you see the largest drops. *** Obviously, these numbers are concerning – Ethiopia is at high risk for importation (given their geographic location) and transmission (given their routine vaccination rates).
The take home is much the same for both Angola and Ethiopia. Both need a renewed emphasis on AFP surveillance. Training materials in both countries are currently undergoing revision and will certainly address this.
Thinking about developing some kind of score for further analysis. Positive associations between project activities and positive vaccination outcomes in all countries support the claim that the project is responsible in part for improvements noted. This information should be disseminated as an illustration of the importance of community-based programming as an essential ingredient to basic primary health interventions, even relatively technological ones such as immunization.
No standard “polio” surveys. Included some questions that didn’t give us useful or reliable data.
1. CORE Group Spring Meeting May 11, 2011 Midterm Evaluation Results CORE Group Polio Project
2. Project Background
3. Project Background Current USAID project cycle: October 2007 – September 2012 Donors: 1999 – September 2012 December 2008 - December 2013
4. Structure of the CGPP
5. CGPP Angola
6. CGPP Ethiopia
7. CGPP Uttar Pradesh, India
8. Project Background
9. Increased routine vaccination rates Increased/ sustained participation in quality SIAs Increased/ sustained reporting of suspected AFP cases OUTCOMES Community-based Social Mobilization Door-to-door Counseling + Child Tracking Group Education Sessions Influencer Involvement Key Surveillance Informants Other Country-specific Activities Capacity-building for Local Health Systems Training Opportunities Technical Assistance for Microplanning Child Tracking Country-specific high-level contributions Campaign Quality Monitoring – Angola Regional Advocacy Workshops – Ethiopia SMNet Partner - India PROJECT ACTIVITIES
10. Presentation Focus Capacity-building for Local Health Systems Training Opportunities Technical Assistance for Microplanning Child Tracking Community-based Social Mobilization Door-to-door Counseling + Child Tracking Group Education Sessions Influencer Involvement Key Surveillance Informants Other Country-specific Activities Country-specific high-level contributions Campaign Quality Monitoring – Angola Regional Advocacy Workshops – Ethiopia SMNet Partner - India PROJECT ACTIVITIES Increased knowledge of and participation in routine vaccination Increased/ sustained community acceptance of and demand for SIAs Increased/ sustained awareness of AFP and proper reporting procedures OUTCOMES
11. Evaluation Questions <ul><li>Has polio and RI immunization coverage in children 12 – 23 months of age in the geographic areas where the CORE Group Polio Project is working improved compared to baseline (2008)? </li></ul><ul><li>Has AFP surveillance in the areas where the CORE Group Polio Project is working improved compared to baseline (2008)? </li></ul>
13. Household Survey <ul><li>Caregivers of children 12 – 23 months of age </li></ul><ul><li>Similar surveys </li></ul><ul><li>30 Cluster Sampling </li></ul><ul><li>Data collection </li></ul><ul><ul><li>Baseline - summer 2008 </li></ul></ul><ul><ul><li>Midterm - summer/fall 2010 </li></ul></ul>Angola Ethiopia India Baseline n 387 2109 605 Design Features One set of 30 clusters Set of 30 clusters in each of 7 states Set of 30 clusters in each of 2 project regions Midterm n 450 887 603 Design Features One set of 30 clusters Set of 30 clusters in each of 3 areas Set of 30 clusters in each of 2 areas
27. Supplementary Immunization Activities <ul><li>ANGOLA </li></ul><ul><li>Solid improvement in SIA performance in a country where successful campaigns are critical </li></ul><ul><li>INDIA </li></ul><ul><li>SIA coverage high at baseline </li></ul><ul><li>Maintaining near complete coverage an accomplishment </li></ul><ul><li>Artifact in final indicator? </li></ul>
30. Acute Flaccid Paralysis Surveillance <ul><li>ANGOLA </li></ul><ul><li>AFP surveillance needs renewed emphasis </li></ul><ul><li>Currently being addressed in revised volunteer training + materials </li></ul><ul><li>ETHIOPIA </li></ul><ul><li>AFP surveillance needs renewed emphasis – not unexpected, but must be addressed for certification </li></ul><ul><li>Also currently being addressed in revised volunteer training + materials </li></ul>
31. Volunteer Contact ASSOCIATIONS - VOLUNTEER/WORKER CONTACT & PROJECT OUTCOMES Door-to-door counseling Group education sessions <ul><li>Positive associations between project activities & vaccination outcomes in all countries </li></ul><ul><ul><li>Ethiopia – strongest associations between outcomes & group sessions </li></ul></ul><ul><ul><li>India – strongest associations between door-to-door counseling and campaign outcome </li></ul></ul><ul><ul><li>Strong associations between project activities and AFP knowledge and attitudes in Angola only </li></ul></ul><ul><ul><ul><li>Reflects de-emphasis in India and concerning findings in Ethiopia? </li></ul></ul></ul><ul><ul><ul><li>Plans for further analysis </li></ul></ul></ul>
32. Questions that didn’t work <ul><li>When was the last vaccination campaign? </li></ul><ul><li>Is it meaningful to remember a date? </li></ul><ul><li>Inconsistent frequency of campaigns. </li></ul><ul><li>Do you believe children can receive polio vaccinations too often? </li></ul><ul><li>Translation problems - ambiguity </li></ul><ul><li>Lifetime doses of polio vaccine </li></ul><ul><li>Recall reliability – inconsistent with other vaccination measures </li></ul>