This document summarizes an analysis of metadata from the CommCare mobile data platform, which is used by hundreds of humanitarian programs worldwide. The analysis examined CommCare usage data to better understand how frontline workers and programs utilize the technology. Key findings included: 1) Frontline workers develop proficiency quickly in their first year but then level off; 2) Worker activity levels remain relatively stable month-to-month; 3) Worker productivity does not follow a normal distribution. The metadata analysis can help programs monitor performance and identify factors that lead to improvement over time.

1. MOBILE TECHNOLOGY
USAGE BY HUMANITARIAN
PROGRAMS:
A METADATA ANALYSIS
Rashmi Dayalu
O P E N
D A T A
S C I E N C E
C O N F E R E N C E_
BOSTON 2015
@opendatasci

2. 1
Mobile technology usage by humanitarian programs:
a metadata analysis
Open Data Science Conference
May 31, 2015
Rashmi Dayalu
Data Scientist
Dimagi, Inc.

3. 2
#ODSC
We are called to work “where our greatest passion
meets the world's greatest need.”
- Frederick Buechner

4. 3
#ODSC
Background

5. 4
Access to humanitarian services is limited in most parts
of the world.

6. 5
#ODSC

7. 6
Frontline Workers (FLWs) are often the primary link
between underserved communities and humanitarian
services.

8. 7
#ODSC
“How can I keep track of my pregnant clients’ medical
information, visit schedules and due dates?” – Seema

9. 8
#ODSC
“How can I show videos of the best agricultural
practices to the farmers in my community?” – Yann

10. 9
Meeting the world’s greatest needs: “We deliver open and innovative technology to
help underserved communities around the world.”

11. 10
#ODSC
Dimagi’s core product

12. 11
#ODSC
 Open source mobile technology platform
 Does not require software developers to configure or deploy mobile
applications
 Can be used on feature phones, androids, tablets, on the web or over
SMS
Image: http://www.ictedge.org/projects/zeprs
X 

13. 12
Data collection Client Counseling
Case management and workflow supportTraining reinforcement and supervision
The result? Stronger healthcare workers
and stronger communities…
#ODSC

14. 13
#ODSC
There are CommCare users in over 40 countries
4001 – 5000
3001 – 4000
2001 – 3000
1001 – 2000
1 - 1000

15. 14
#ODSC
CommCare is used across multiple sectors and subsectors

16. 15
#ODSC
• CommCare’s cloud server hosts data from hundreds of
humanitarian programs.
• We are using CommCare metadata to ask a variety of questions
that can aid programs and FLWs in their goals.
http://noble1solutions.com/wp-content/uploads/2014/06/what-is-big-data.jpg

17. 16
#ODSC
BREADTH
D
E
P
T
H
Program A Program B …. Program “X”
FLW 1
FLW 2
FLW 3
…
FLW “N”
How do programs and FLWs perform across the board?
Ismyprogramperformingwell?
AremyFLWsperformingwell?

18. 17
Measuring FLW and program performance
using CommCare metadata

19. 18
Form creation is the most basic unit of CommCare usage
#ODSC

20. 19
CommCare form metadata
FLW ID CASE ID
Form start/end date
Form start/end time
#ODSC

21. 20
#ODSC
Dimagi’s data platform

22. 21
#ODSC
Cumulative # form submissions and # new cases registered with Commcare:
all programs, 2010 - 2014

23. 22
#ODSC
1. How quickly do FLWs develop proficiency
with CommCare?
Analysis by: Jeremy Wacksman

24. 23
#ODSC
We looked at 634 workers who used CommCare for at least one year and
were active for at least 10 months of their first year.
Q1
Q2
Quarterly
range
Median
change
Q1 – Q2 + 22.9%
Q2 – Q3 + 1.9%
Q3 – Q4 + 0.0%
Q3
Q4

25. 24
#ODSC
2. Are FLW activity levels stable from month to
month?

26. 25
#ODSC
Intra-user consistency:
• After the 6 month adoption period, do FLWs
maintain stable levels of CommCare activity?
• We calculated the Pearson correlation coefficient for
all pairs of consecutive calendar months for
individual FLW activity levels

27. 26
#ODSC
Intra-user consistency for % active days and # forms (663 FLWs)
r = 0.68r = 0.70

28. 27
#ODSC
Programs can use the hypothesis of intra-user consistency to
monitor unexpected changes in FLW activity levels:
e.g. (1) Are FLWs less active during certain seasons or
months of the year?
N = 5,303 monthly observations (from health programs in India)

29. 28
#ODSC
(2) Do FLWs show decreased activity levels prior to
attrition in CommCare activity (inactivity >= 90 days)?
N = 252 FLWs with at least one CommCare attrition event

30. 29
#ODSC
3. Do FLW activity levels follow a bell curve?
Analysis by: Mengji Chen

31. 30
#ODSC
1. Boyle, E., Aguinis, H. “The Best & the Rest: Revisiting the Norm of Normality and Individual Performance”, Personnel
Psychology, 2012, 65, 79-119.
2. Image from: http://www.marin.edu/~npsomas/Lectures/Ch_1/Section_03.htm
Normal distributions are the most commonly held assumption in performance
metrics1. Is this assumption valid for CommCare FLWs?

32. 31
#ODSC
Distribution of % active days/month for 12 programs with the most active months

33. 32
#ODSC
Boyle, 2012. Personnel Psychology
Programs have larger number of FLWs that are either underperforming or hyper-
performing.
Workloads, performance ranking, training and compensation cannot assume the
norm of normality.

34. 33
#ODSC
4. Do FLWs use CommCare in real-time while
interacting with their clients?

35. 34
#ODSC
There is no way to confirm real-time data collection by FLWs using
metadata, but we can flag visit data that was unlikely to have been
entered in real-time:
1. Batch entry – visits entered consecutively in quick succession (e.g.
with < 10 minutes between visits)
2. Visit duration (e.g. < 1 minute)
3. Visit time of day (e.g. visits started at night, between 6pm – 6am)

36. 35
#ODSC
Batch visits (%) by program
Proportion of batch visits from 30 maternal and child health programs worldwide

37. 36
#ODSC
Programs with unexpectedly large daily visit volumes revealed that:
(1) Patient data was often uploaded automatically via CommCareHQ -
CommCare’s web interface (e.g. maternal registrations)

38. 37
#ODSC
(2) Manual batch entry might actually be part of regular work flow for FLWs in
clinical settings (e.g. immunizations, child anthropometrics, etc.)
We looked at batch entry rates for 9 programs that had at least one “travel visit”
component built into their apps.

39. 38
#ODSC
In conjunction with batch entry, visit duration and visit time of day can be used to
flag visit data that was unlikely collected in real-time.
Visit duration (Mood’s Median Test):
Batch visits for programs A, B and C combined were ~half the duration of non-batch
visits (median duration of batch visits = 3.8 minutes, median duration of non-batch
visits = 7.7 minutes, Z = 5.35, p < 0.001).
Visit time of day (Chi-square Test):
Batch were more likely to have been recorded in the night
(% night non-batch visits = 16.5% and % night batch visits = 20.8%, 2 = 178.99, df = 1, p
<0.001).

40. 39
#ODSC
5. How long do programs use CommCare for?
Analysis by: Patrick Keating

41. 40
#ODSC
Sustainable use of CommCare is evidence for CommCare’s value. Of 306 programs,
how many were still active in Q4 2014?
Distribution of # programs by # active months and activity status in Q4 2014

42. 41
#ODSC
176 (57.5%) programs stopped using CommCare for at least 3 months. Of those,
43% restarted their CommCare usage, though restart rates are dependent on the
age of the program.

43. 42
#ODSC
Programs with more active FLWs were more likely to be active through 2014
This could mean that programs with smaller numbers of users have limited
resources and sometimes cannot continue their activities - regardless of how
effective CommCare is.

44. 43
#ODSC
6. Which programs are improving over time?
Algorithm developed by:
Dag Holmboe
Dimagi’s Data Science Advisor
Founder of Klurig Analytics
http://www.kluriganalytics.com

45. 44
#ODSC
Detecting improvement can help us concretely identify the programmatic factors
that led to the improvement.

46. 45
#ODSC
Preliminary validation: Program #60
Performance feedback to FLWs in the middle of the year could have
contributed to the continued improvement (beyond first 6 months).

47. 46
#ODSC
Some future investigations:
1. Do 20% of FLWs submit 80% of the data?
2. Do programs that use supervisory tools have
the most active FLWS?
3. Is CommCare activity correlated with socio-
economic indicators (GNP, literacy rates,
corruption index, etc.)?
4. How do CommCare crashes affect user
behavior?

48. 47
Thank you!
For questions or research opportunities, please contact:
Rashmi Dayalu
rdayalu@dimagi.com
#ODSC

49. 48
#ODSC
Activity metric by FLW
per calendar month
Definition
1. # forms Total number of electronic forms submitted
2. # visits Total number of visits made to all cases
3. # cases Total number of unique cases visited (either registered
or followed up)
4. # cases registered Total number of unique cases registered
5. # cases followed-up Total number of unique cases followed-up
6. % of active days Percentage of days in the month during which the CHW
submitted data
7. Total duration of
CommCare use (min)
Cumulative time using CommCare, i.e. sum of all visit
durations
CommCare activity metrics - Aggregated by calendar month
per FLW

50. 49
#ODSC
Calculating breakpoints:
• Rolling window of mean FLW activity levels over the program
lifetime
• If window mean is at least 3 SD’s higher then the previous window, it
is a candidate breakpoint
• t-test of means between the windows confirms the statistical
significance of the breakpoint