Is it possible to create a generic mobile app to manage and monitor community scorecard activities? This presentation summarises research based on work in Mozambique.

1. The challenges of implementing generic
web and mobile apps for managing and
monitoring community scorecards and
social audits: lessons from Mozambique
SAMEA Conference
15th October 2015

2. Introduction
• Kwantu is a Cape Town based social
enterprise that aims to make M&E easier
• We make affordable mobile and web apps
to manage and monitor activities
• These connect to an fully-featured
monitoring and evaluation system

3. Research rationale

4. Research rationale (1)
• Is it possible to create a standardised
M&E app for a specific type of project or
activity?
• What flexibility is needed to address
individual requirements?
• What benefits can standardisation bring?

5. Research rationale (2)
• M&E apps are normally created for a
specific organisation or programme
• Done in isolation from others
implementing similar activities or projects
• Rare to explore the similarities of the
requirements across different
organisations

6. Research rationale (3)
• This ensures that systems meet the
needs of the organisation
• But, it costs more to implement and
maintain
• And, it makes data sharing across
organisations harder
• And, opportunities to learn from others
are reduced

7. Research rationale (4)
• Attempts to create shared M&E apps
usually focus on indicator level
• This research explores the challenges of
creating shared apps that focus on
monitoring the activity level
• Better quality data as we can aggregate
up to calculate indicators

8. Research context

10. Research context (1)
• Selected Mozambique and social
accountability as a research context
• First significant social accountability
project in Mozambique started in 1996
• Focused on citizen engagement around
debt relief process

11. Research context (2)
• Number, scale and type of social
accountability projects growing
significantly
• Tools like community dialogues, social
audits, citizen report cards and
community score cards are now widely
used by NGOs

12. Research context (3)
• Several NGOs now implementing at scale
• 100+ service providers a year
• Large scale activities using same social
accountability tools in same communities
each year

13. Research context (4)
• Increased scale brings challenges related
to management:
• Planning (who is working where?)
• Coordination (who should we be
collaborating with and where?)
• Management (how do we schedule and track
the meetings and workshops needed to
engage citizens?)

14. Research context (5)
• Scale also magnifies M&E challenges:
• Collecting large quantities of data on
activities
• Prone to double-counting and data
transcription errors
• Difficult and time consuming to aggregate
and analyse data

15. Research context (6)
• Policymakers also face challenges:
• Impact takes time to see and is difficult to
measure and attribute
• Difficult to assess economy, effectiveness
and efficiency (value for money) of different
interventions
• Data from different implementers is not
easily comparable

16. Research context (7)
• Lessons from private sector franchising?
• Franchise owner agrees a set of
standardised business processes
• These include standards on which
information must be collected, how and
when
• Shared technology to manage the process
and collect and share the data

17. Research context (8)
• Lessons from private sector franchising?
• Is this approach transferable to the social
accountability sector?
• Or will it limit implementers too much in how
they manage and monitor activities?

18. Research partners

19. Research partners (1)
• Citizen Engagement Programme (CEP)
• Working in 4 provinces (Gaza, Manica,
Nampula and Zambezia)
• Implementing community scorecards across
100+ schools and health facilities
• Five year programme funded by DFID,
DANIDA and Irish Aid

20. Research partners (2)
• Centro de Aprendizagem e Capacitação
da Sociedade Civil (CESC)
• Working in 4 provinces (Gaza, Cabo
Delgado, Tete and Zambezia)
• Implementing community scorecards across
50+ schools, health facilities and
municipalities
• Ongoing work funded by several donors

21. Research partners (3)
• N’weti Health Communication (CESC)
• Working in 4 provinces (Gaza, Nampula,
Maputo and Maputo City)
• Implementing community scorecards across
80+ health facilities
• Ongoing work funded by several donors

22. Research partners (4)
• Concern Universal
• Working in 3 provinces (Cabo Delgado,
Niassa and Zambezia)
• Implementing social audits across 6
municipalities
• Ongoing work funded by SDC

23. Research partners (5)
• Estamos
• Working in 1 province (Niassa)
• Implementing social audits across 3
municipalities
• Ongoing work funded by several donors

24. Research partners (6)
• Research focuses on community
scorecard activities implemented by CEP,
CESC and N’weti
• More data needed to compare social audit
approaches (additional research partner
needed)

25. Research partners (7)
• Community scorecards are a tool to
facilitate citizen monitoring of the quality
of service delivery
• Based on scorecards where citizens:
• Define performance issues that matter to
them
• Score them against satisfaction criteria

26. Methods

27. Methods (1)
• Analysing and documenting partner
requirements:
• Process – What are the sequence of steps
followed to implement the activity?
• Data – Which data collection forms are
needed to collect data?
• Taxonomies – How will data be coded for
analysis?

28. Methods (2)
• Process
• Break activities down into a series of steps
• Steps are linked to key points where
management information is needed or data
reviews are required
• For each step we documented:

29. Methods (2)
• Pre-conditions necessary before the step
begins
• Role player responsible for the step
• Which data collection forms must be
completed
• Which tasks must be completed
• Guidance for field staff
• Which steps the process may move to next

30. Methods (3)
• Data
• Work backwards from data needed for
management and M&E
• Define data collection forms needed to
gather this data
• Link them with relevant steps in the process
when the data can be collected

31. Methods (4)
• Taxonomies
• Which vocabularies will be used to code
data?
• Which forms are these used on?
• What analysis are these needed for?

32. Methods (5)
• Create app based on documentation:
• Configure taxonomies
• Configure forms and link to taxonomies
• Configure workflow and link to forms
• Configure reports to aggregate data
• In use for over a year
• Mobile version coming in December

33. Methods (6)
• Comparative analysis of partner
documentation
• What are similarities and differences
between:
• Process steps
• Forms
• Taxonomies

34. Methods (7)
• Limitations:
• Sample of three NGOs
• Not selected randomly
• Not representative data
• Can draw only preliminary conclusions

35. Results

36. Results (1)
• Process comparison:
• All three partners use similar steps to track
implementation
• Names differ
• Some include sign-off / review steps
• Some include additional steps that help track
specific stages in more detail

37. Results (2)
Planning
Register
groups
Groups
identify and
score service
issues
Dialogue
with service
provider
Agree joint
action plan
Monitor
action plan

38. Results (3)
• Need flexibility to:
• Re-name steps in the process
• Add new steps
• Specify sign-off steps
• Define guidance for field staff

39. Results (4)
• Following core forms documented:
• Project registration form
• Facility survey form
• Group registration form
• Scoring form
• Action plan form

40. Results (5)
• Following optional forms documented:
• Contact
• Outcome
• Meeting
• Action plan monitoring form

41. Results (6)
• Types of forms used are very similar
• Questions within forms also very similar
• Need to consider:
• How to collect data needed for planning and
value for money analysis?
• How to define sector specific questions?
• How to select which forms to use?

42. Results (7)
Taxonomy CEP N’weti CESC
Province Yes Yes Yes
District Yes Yes Yes
Facilities Yes Yes Yes
Group type Yes Yes Yes
Priority Yes Yes Yes
Level Yes Yes Yes
Common issues Yes Yes Yes
Scorecard type Yes Yes Yes
Donor No Yes Yes
Year No Yes Yes
Project No Yes Yes
Partner No No Yes

43. Results (8)
• Similar taxonomies in use
• Work need to standardise some:
• Group types
• Common issues
• Option needed to add partner specific:
• Donors
• Partners, etc.

44. Conclusions

45. Conclusions (1)
• Feasible to create a generic, standardised
app for managing and monitoring
community scorecard projects
• But, first need:
• Flexibility to adapt app
• Data standards
• Data sharing mechanism

46. Conclusions (2)
• Flexibility needed for each partner to:
• Adjust workflow (add and re-name steps,
define review/sign-off steps)
• Select forms (choose from a library of forms
to select which they wish to use)
• Edit forms (add in additional questions)
• Edit taxonomies (create and modify
taxonomies that they need)

47. Conclusions (3)
• Set-up wizard needed:
• Take standardised app and guide partner
through types of changes that they can make
• Maintain consistency and data standards
where it counts
• Enable flexibility in other areas

48. Conclusions (4)
• Leadership needed:
• Agree on common taxonomies (national
level?)
• Group types (and definitions for them)
• Common issues
• Satisfaction questions
• Job for Government, UN and donors?

49. Conclusions (5)
• Leadership needed:
• Agree on common data standards needed to
share information
• Project registration form:
• Facility name
• Implementer
• GIS coordinates (from facility taxonomy)
• Start and end date
• Budget categories

50. Conclusions (6)
• Data sharing mechanism:
• With common data standards can begin
sharing data
• However, doing so manually is error prone
and time-consuming
• Mechanism needed to facilitate automated
data sharing based on governance controls
• Prototype in 2016

51. Implications

52. Implications (1)
• Potential new approach to M&E
• Based on shared data definitions linked to
types of activities or projects
• Balances standardisation with flexibility
• Provides the building blocks for M&E
apps

53. Implications (2)
• Implementers:
• Lower costs (shared cost of setting up and
maintaining technology)
• Access and adapt standard data definitions
for common types of activities
• Share planning data to coordinate projects
and improve linkages
• Potential to share knowledge and improve
communication among implementers

54. Implications (3)
• Donors and Government:
• More standardised data means activities are
more comparable
• Possible to aggregate data for comparative
analysis (value for money, spatial distribution
of projects etc.)
• Automated data sharing brings efficiencies to
reporting processes (no more reports…)

55. Implications (4)
• Broader implications:
• Relevance to community scorecard projects
in other countries?
• Relevance to other types of activities and
projects?
• Early indications are positive, but need more
research to validate these

56. Rob Worthington
rob@kwantu.net
www.kwantu.net
Visit blog.kwantu.net and subscribe to follow
this work