This presentation was made by Dr. Sumathi Swaminathan and Mr. Jithin Sam Varghese (St. John’s Research Institute) in the session on 'Assessing coverage and performance of nutrition interventions: Research experiences from across India' at POSHAN's "Delivering for Nutrition in India - Learnings from Implementation Research" conference, November 9–10, 2016 , New Delhi.

1. A TOOL TO RAPIDLY ASSESS GAPS IN IN
DISTRICT-LEVEL COVERAGE OF
NUTRITION INTERVENTIONS IN
MULTIPLE DOMAINS
Sumathi Swaminathan and Jithin Sam Varghese
1

2. All standard surveys have common problems
which need to be addressed
Interview Fatigue
Cost
Data capture and monitoring
Analysis time
Field management
Data falsification
O
“Rapid” surveys
tend to address
pieces of the
puzzle, never the
whole.
2
O
O
O
O
O
Possible IssuesO

3. The RAPID Short survey tool was derived from a
“checklist” approach for ease of administration
and more robust data collection
Boeing Aircraft
Checklist*
WHO Safe Surgery
Checklist
SYSRA
WHO Safe
Childbirth
TESSA*
1935 2008 2012 2013
HISTORY OF THE PUBLIC HEALTH CHECKLIST- SOME MILESTONES
3
*Not directly related to health

4. The RAPID Short survey tool involves a “checklist”
questionnaire followed by a “deep-dive” survey
Boeing Aircraft
Checklist*
WHO Safe Surgery
Checklist
SYSRA
WHO Safe
Childbirth
TESSA*
1935 2008 2012 2013
HISTORY OF THE PUBLIC HEALTH CHECKLIST- SOME MILESTONES
4
*Not directly related to health*Not directly related to health
Mother centric programs
LactatingPregnant Child under 2
At Delivery Post deliveryBefore delivery
Breastfeeding
counselling
Janani Suraksha
Yojana
Understood Followed
Why did you
not follow?
Why was it
not provided?
If Yes
CHECKLIST
DEEP-DIVE
Would be “deep-dived” if
found significant based on
analysis in terms of
determinant/gap
If No
If No

5. Suppose, a LW is asked whether she was provided
breastfeeding counselling during her delivery
Boeing Aircraft
Checklist*
WHO Safe Surgery
Checklist
SYSRA
WHO Safe
Childbirth
TESSA*
1935 2008 2012 2013
HISTORY OF THE PUBLIC HEALTH CHECKLIST- SOME MILESTONES
5
*Not directly related to health*Not directly related to health
Mother centric programs
LactatingPregnant Child under 2
At Delivery Post deliveryBefore delivery
Breastfeeding
counselling
Janani Suraksha
Yojana
Understood Followed
Why did you
not follow?
Why was it
not provided?
If Yes
CHECKLIST
DEEP-DIVE
Would be “deep-dived” if
found significant based on
analysis in terms of
determinant/gap
If No
If No

6. Assuming Poor Breastfeeding Counselling came out
significant in analysis, only participants who answered
“No” will be asked reasons
Boeing Aircraft
Checklist*
WHO Safe Surgery
Checklist
SYSRA
WHO Safe
Childbirth
TESSA*
1935 2008 2012 2013
HISTORY OF THE PUBLIC HEALTH CHECKLIST- SOME MILESTONES
6
*Not directly related to health*Not directly related to health
Mother centric programs
LactatingPregnant Child under 2
At Delivery Post deliveryBefore delivery
Breastfeeding
counselling
Janani Suraksha
Yojana
Understood Followed
Why did you
not follow?
Why was it
not provided?
If Yes
CHECKLIST
DEEP-DIVE
Would be “deep-dived” if
found significant based on
analysis in terms of
determinant/gap
If No
If No

7. If Breastfeeding Counselling was provided but not
followed, and that was significant, then those
participants will be asked reasons for the same
Boeing Aircraft
Checklist*
WHO Safe Surgery
Checklist
SYSRA
WHO Safe
Childbirth
TESSA*
1935 2008 2012 2013
HISTORY OF THE PUBLIC HEALTH CHECKLIST- SOME MILESTONES
7
*Not directly related to health*Not directly related to health
Mother centric programs
LactatingPregnant Child under 2
At Delivery Post deliveryBefore delivery
Breastfeeding
counselling
Janani Suraksha
Yojana
Understood Followed
Why did you
not follow?
Why was it
not provided?
If Yes
CHECKLIST
DEEP-DIVE
Would be “deep-dived” if
found significant based on
analysis in terms of
determinant/gap
If No
If No

8. The “checklist” type RAPID short survey would
be analyzed by machine learning algorithms only
after coverage of adequate sample size
Survey
Data +
Anthro
Algorithm
provides results
in ~3-5 mins
8

9. Algorithms not requiring human intervention
performed as well as logistic regression in exhaustive
survey for adolescent girls
9
Random Forests to identify major associated
determinants
1

10. Random Forests have the following advantages
10
Considers interaction effects
C
Gives a measure of ranked-variable importance via
internal cross-validation to minimize overfitting
B
Handles “order-effects” where a variable is
influenced by those introduced before it in the
model as well as those after it
A

11. In general, Random Forests identify more variables
than Logistic Regression due to nature of the
algorithm
11
Nandurbar
• Avoid school
• Health seeking behaviour
• Citrus fruit consumption
• Health and hygiene
practices
• Avoid school
• Health Seeking Behaviour
• Citrus fruit consumption
• Health and hygiene
practices
• Knowledge of state
language
• Beneficiary programs
• Mother’s education
• Dietary diversity
• Citrus fruit consumption
• Mother’s education
• Dietary diversity
• Citrus fruit consumption
• Avoiding school
• Wealth quintile
• Tea consumption
Sirohi Logistic Regression Random Forests

12. Additionally, internal cut-offs to identify variables
with low variability (very high/ very low prevalence)
in data
12
Internally deliberated coverage cut-off scores to
identify poorly functioning services/domains-
“Catch-all”
2

13. 13
Prevalence of variables were similar (95% confidence
interval) across both RAPID (n=251 ) and Exhaustive (n=
759) questionnaires despite smaller sample size for
RAPID “checklist” questionnaire for Adolescent girls
SOME VARIABLES IN RAPID SHORT SURVEY :
1-% who consume atleast 6 food groups atleast for 4 days in a week, 2-% who have consumed more than 3 Citrus
Days, 3-% currently going to school, 4-% receiving midday meals among those going to school, 5-% consuming
midday meals among those receiving, 6-% provided IFA, 7-% receiving Deworming, 8-% who did not suffer from
diarrhoea in last 1 month, 9-% who are literate, 10-% using mosquito prevention, 11-% who go to a government/
allopathy or NGO hospital for treatment, 12-% who use Iodized salt
Nandurbar
RAPID Exhaustive

14. 14
For pregnant women also, prevalence of parameters in
RAPID (n=34 ) and Exhaustive (n=324) were similar.
However, the confidence interval was quite wide due to
very small sample size.
SOME VARIABLES IN RAPID SHORT SURVEY :
1- % who have been provided IFA or FA (Reported),2- % who have consumed non-zero days of IFA/FA,3- % who have received
deworming,4- % who have registered in ANC clinic,5- % whose weight was measured among those who visited the ANC clinic non-zero
times or who have ANC card,6- % who have been provided Breastfeeding counselling among Trimesters 2 and 3,7- % who consume
atleast 6 food groups atleast for 4 days in a week,8- % ICDS Received,9- % ICDS Consumed all by self among those who received and
did not share,10- % who have consumed more than 3 Citrus Days,11- % who did not consume Paan,12- % whose gravidity is not more
than 3,13- % having improved water source,14- % using improved water purification method,15- % using mosquito prevention,16- %
who are willing to pay a nominal amount for health insurance,
RAPID Exhaustive
Nandurbar

15. 15
Prevalence in children between 2 to 5 years were not
comparable between RAPID (n=268) and Exhaustive (n=
845 )for many variables due to sampling issues
SOME VARIABLES IN RAPID SHORT SURVEY :
1- % consume ≥6 food groups for atleast 4 days in a week,2- % consumed > 3 Citrus days,3- % received any ICDS supplementary
nutrition,4- % consumed supplementary nutrition alone among those who received,5- % those who have been consuming Vitamin
A in last 1 month,6- % who have been provided IFA,7- % who have received deworming in last 6 months,8- % children whose weight
was checked in last 1 month,9- % children whose height was checked in last 1 month,10- % children who did not suffer from
diarrhea in last 1 month,11- % children who consumed ORS among those who had diarrhea,12- % who are literate,13- % who wash
hands with soap after defecation,14- % who wash hands with soap before preparing food,15- % who take child for treatment when
sick,16- % who use iodized salt,
RAPID Exhaustive
PENDING
Nandurbar

16. 16
The “deep-dive” survey is tailored to ask the
“why”s to those households suffering from
problems identified via the “checklist”
Survey
Data +
Anthro Deep-dive
List
The deep-
dive
survey will
be tailored
specific to
each
household
16

17. 17
The “deep-dive” survey is tailored to ask the
“why”s to those households suffering from
problems identified via the “checklist”
Survey
Data +
Anthro Deep-dive
List
The deep-
dive
survey will
be tailored
specific to
each
household
17

18. 18
The “deep-dive” survey is tailored to ask the
“why”s to those households suffering from
problems identified via the “checklist”
Survey
Data +
Anthro Deep-dive
List
The deep-
dive
survey will
be tailored
specific to
each
household
18

19. 19
RAPID short survey takes about 20 days to complete due
to more interviews per day and survey optimization
Item Exhaustive RAPID Difference
Time for survey 70 days 20 days 50 days saved
* Excluding fixed costs; L- Rupees in Lakhs

20. 20
RAPID short survey requires ~ ₹ 15 lakhs* and 20 days
for completing a district utilizing associated survey
optimization tools
Item Exhaustive RAPID Difference
Time for survey 70 days 20 days 50 days saved
Survey and training
costs
₹ 28-32L ₹ 14-17L ₹ 14L saved
Fixed costs
(e.g. Equipment)
₹ 4L ₹ 8-10L ₹ 4-6L additional
CAPI development and
data analysis
60 days
(₹ 7-10L)
7-10 days
(~₹ 1 L)
~50 days &
> ₹6 L saved
* Excluding fixed costs; L- Rupees in Lakhs
The cost of any survey is driven by number of man-days. If we reduce this, we could
reduce costs. The ideal way is to conduct more (but reliable) interviews per day

21. 21
RAPID short survey could aid policy makers and
NGOs save time and money using an array of tools!
21
The RAPID short questionnaire could be supported by an expert committee to
further develop this tool for other domains, as well as include a cost component*
for interventions identified
1
*The cost component would aid decision makers identify which intervention would
aid in maximal benefit

22. 22
Plug-and-play modules for RAPID surveys
The modules could be open-sourced or available for use at a small
licensing fee
We intend to develop plug and play modules any
survey team could use to optimize their survey
22
2

23. 23
Plug-and-play modules for RAPID surveys
Tool using existing software such as ODK* to help
survey teams build checklist based questionnaires
Short
Questionnaire
Data Analysis
Route
optimization and
scheduling
Deep-dive
questionnaire
Data anomaly
detection
2
23
Methodology
could be
adopted for
other study
designs;
Customizable
tool to develop
questionnaires
*Open Data Kit survey tool

24. 24
Plug-and-play modules for RAPID surveys
Machine learning algorithms such as Random
Forests for quick analysis of data
Short
Questionnaire
Data Analysis
Route
optimization and
scheduling
Deep-dive
questionnaire
Data anomaly
detection
2
24
Analysis to be
machine driven
with
appropriate
domain
weights and
domain-specific
cutoffs

25. 25
Plug-and-play modules for RAPID surveys
Field managers could determine where to send
their teams real-time
Short
Questionnaire
Data Analysis
Route
optimization and
scheduling
Deep-dive
questionnaire
Data anomaly
detection
2
25
Could help save
more time

26. 26
Plug-and-play modules for RAPID surveys
Anomaly detection algorithms could be built in at
server-side to identify interviewers/teams who are
functioning poorly and may need retraining
Short
Questionnaire
Data Analysis
Route
optimization and
scheduling
Deep-dive
questionnaire
Data anomaly
detection
2
26
Error detection
at multiple
levels for more
robust data

27. 27
Plug-and-play modules for RAPID surveys
The “deep-dive” questionnaire is important to
identify the “why”s
Short
Questionnaire
Data Analysis
Route
optimization and
scheduling
Deep-dive
questionnaire
Data anomaly
detection
2
27
Exhaustive
question
repository;
costs 1/3rd of
RAPID via
optimization

28. 28
The deep-dive round could be further developed to include the related supply
side* determinants
3
We hope to combine deep-dive questionnaire
along with related supply side indicators for
reliable decision making
28
* As of now, we have exhaustive supply side questionnaires for ASHA, Anganwadi, Sub
Centre, Primary Health Centre, Community Health Centre/Rural Hospital, Sub
Divisional Hospital/District Hospital

29. Ongoing/Completed Pending
In summary, RAPID short survey promises an
effective way of optimizing the survey process
29
Interview Fatigue
Cost
Data capture and monitoring
Analysis time
Field management
Data falsification
P
P
P
P
!
!
Short questionnaire
Reduced survey time lowers cost
Easy to deploy android app
Route optimization and scheduling
Machine driven classification
Anomaly detection algorithms
P !

30. Way forward
• Validation of RAPID checklist:
– Full sample size (presently, it was on 1/10th households)
– Algorithm validation across domains for additional proof-of-
concept (program evaluation, agriculture, SAM-MAM etc)
• Development:
– Fine-tune “checklist” for different regions
– Link deep-dive repository in app
– Route optimization and Anomaly detection algorithms to be
built in
• Consultation:
– Domain importance weights for Analysis automation
– Domain specific coverage cut-offs for “catch-all”
30