MobileActive 08: Mobile metrics and Evaluation Patty Mechael Peter Benjamin

OUTLINE OF THE SESSION Introductions of all Metrics & evaluation framework Case study: “Mobile 4 HIV” @ SA Group work Report back Next steps  Mobile Metrics Work Group?

OUTLINE OF THE SESSION

Introductions of all

Metrics & evaluation framework

Case study: “Mobile 4 HIV” @ SA

Group work

Report back

Next steps  Mobile Metrics Work Group?

Introductions Name Organisation Interest in mobile metrics? (Research question, techniques, sector)

Name

Organisation

Interest in mobile metrics? (Research question, techniques, sector)

Counting SMS is not evaluation! Input - Budget, tech, people … Output - Usage, User info, TGI, SMS … Outcomes - Action from info Impacts - Wider social change

Input - Budget, tech, people …

Output - Usage, User info, TGI, SMS …

Outcomes - Action from info

Impacts - Wider social change

Case study: “Mobile 4 HIV” Project in SA to use mobile for range of services for HIV sector Working with HIV-related organisations: Treatment Action Campaign, Soul City, DTHF, HIVAN, SANAC Low costs, language, low-tech, local Pilots, content, develop tech, research

Project in SA to use mobile for range of services for HIV sector

Working with HIV-related organisations: Treatment Action Campaign, Soul City, DTHF, HIVAN, SANAC

Low costs, language, low-tech, local

Pilots, content, develop tech, research

Start with stated aim of the project – don’t be blinded by the tech. Adherence indictor, Clinic stats, patient QoC Adherence Loss to follow-up Patient – clinic link Adherence indicator, focus group. Uptake of services Improved positive living, adherence Messaging: Treatment Behaviour change indic. Self-report. Uptake of services Ideally – reduced incidence. Behaviour change Messaging: Prevention Eval Tools (e.g.) Goal Area

Eval Tools (e.g.) Goal Area Self-perception of reduced isolation, Improved question response Reduced isolation, self-help, respond to need Peer-peer, councelling Chosen indicators (eg branch reports) Feeling of inclusion and organisational capacity Improved info flow inside an organisation Org capacity Length of time to complete, data validation techniques Improved data collection (data quality, faster) M&E

Cellphone as evaluation tool Also, cellphones can be used as a tool for evaluation (data collection) Can random sample users and call / SMS back (ethics, opt-in and prior approval) Fairly easy to do longitudinal and follow-up studies Quiz on cellphones (esp SMS or USSD) for pre- and post-questions

Also, cellphones can be used as a tool for evaluation (data collection)

Can random sample users and call / SMS back (ethics, opt-in and prior approval)

Fairly easy to do longitudinal and follow-up studies

Quiz on cellphones (esp SMS or USSD) for pre- and post-questions

Other questions Social meaning of messages (not neutral, language, literacy). Ethnography & sociology issues Pulling out contribution of mobile to intervention (confounding variables) Scientific studies: randomised, double blind, controlled

Social meaning of messages (not neutral, language, literacy). Ethnography & sociology issues

Pulling out contribution of mobile to intervention (confounding variables)

Scientific studies: randomised, double blind, controlled

Group discussion Business case, ROI Use of mobile as research tool Assess impact, monitoring & eval Topic areas: HIV, Health, water & an, education, youth, pricing, finance, universal birth registration

Business case, ROI

Use of mobile as research tool

Assess impact, monitoring & eval

Topic areas: HIV, Health, water & an, education, youth, pricing, finance, universal birth registration