gLOCAL 2021 WhatsApp Analysis webinar presentation KhulisaMS

7/9/2020 1
Using as a data
source for evaluation
gLocal Evaluation Week: June 1, 2021
Sharing local and global M&E knowledge
Jennifer Bisgard, Mosa Leteane, Carole Metekoua & Heather Dixon

AGENDA…
7/9/2020
2
1. Introduction to Khulisa
2. Welcome to Participants from around the world!
3. Ice breaker
4. Why Khulisa used WhatsApp data (includes Q&A) - Jennifer
5. Break
6. Reflections and Scenarios - Mosa
7. Technical how to analyze data - Carole
8. Questions and Comments - Jennifer
#gLocalEval2021 and Tag @KhulisaMS

JENNIFER BISGARD
3
Ms Jennifer Bisgard co-founded Khulisa
Management Services in 1993. An expert
in monitoring and evaluation (M&E) and
organizational development, she leads
evaluations and capacity building
assignments in Education, Power,
Democracy and Governance sectors.

MOSA LETEANE
4
Ms Mosangoaneng Leteane joined Khulisa in
2020 as an Emergent Evaluator. As a Mandela
Rhodes alumna, she holds a Masters in
International Studies and Human Rights.
Note, she has a weakness for Cinnabons with
extra frosting!

CAROLE METEKOUA
5
Ms Carole Metekoua joined Khulisa in 2019
as an Data Scientist. She has a Masters in
Public Health from Manchester Metropolitan
University and an Msc in Public Health
Informatics from the University of the
Witwatersrand. She joins us from Cameroon!

HEATHER DIXON
6
Ms Heather Dixon joined Khulisa in 2019 as
an Emergent Evaluator. She works across
projects; particularly in the sectors of
education, human rights and electricity/energy.
She is passionate about equity, social justice,
her two cats and cooking!

7
335 participants registered for this
session, here is a little about you…
188 147

335 participants
from 63 countries

You are…
You are from…
Evaluation Practitioner
Researcher (academia)
Other
Independent
Student
Government employee
NGO
Private Sector
Academic Institution
Other
Government
Multilateral Organization

10
Menti Ice breaker
www.menti.com
CODE: 1240 4233

ChatBox!
Let us know in the ChatBox how you (or
your project/program) currently use
WhatsApp!

6/2/2021 Contract Number: 72067418D00001, Order Number: 72067419F000015 12
About the Reading
Support Project
Design and Implementation Evaluation
PIXABAYSTOCKSNAP

This US $5.5 million project targeted
• Approximately 255 low income rural and urban schools
• 1,530 Grade 1, 2, and 3 teachers
• Both Setswana and English as a First Additional Language
Through:
Teacher training Learner and teacher
support materials
School Management
Team training
14 Coaches and
2 Head Coaches
Classroom libraries

Scale up of the Early Grade Reading
Study I which piloted coaching, designed
as Randomized ControlTrial
RSP Purpose: to improve the quality of
teaching and learning in the classroom
and, in turn, improve early grade reading
6/2/2021 14
RSP
Treatment Control
RSPTreatment & Control schools
in two Districts
“Coaching & SMT”
67 schools
“Coaching”
68 schools
“Materials & Training”
122 schools
34 overlay classroom libraries 32 overlay classroom libraries 33 overlay classroom libraries
Lesson Plans
Train Subject Advisors
Cluster based training for teachers on Lesson Plans
Coaching
SMT & Principal

Evaluations
Design Evaluation
November 2019
Formative Evaluation
March 2020
Summative Evaluation
May 2021
COVID-19 Disruption
March 2020 to Sept 2020

Evaluation Timeline
6/2/2021 USAID DBE RSP MEETING 16
Fieldworker
Training
August
Coach
Shadowing &
Pilot Testing
September
Tool Revision Coach &
Teacher
Training
Observation
October
Fieldworker
training &
Simulation
October – November
Data Collection
at 60 Schools
Coach & Teacher
Training Observation
October 2020
Coach Shadowing
November 2020
Tool Revision
November – December
2020
Data Collection
at 59 Schools
2019
2020
Fieldworker training
& simulation
September - November 2020

DATA COLLECTION
Stakeholder
interviews
Teacher focus
groups
Classroom
observation
Classroom library
assessment
Workbook analysis Teacher survey Principal & SMT
Interviews
School-level Data
Quality Assessment

COVID-19 Pivot
6/2/2021 18
RSP instructed Coaches to use WhatsApp due to COVID-19
Coaches reported that they used WhatsApp to respond to
questions from teachers, to send out activities and other
education materials to assist teachers
“I have WhatsApp groups with my coaching teachers and my
LTSM schools teachers. I sent them a message every day.”
(Coach 9)

Teacher Survey
But, RSP collected no monitoring data on this “virtual coaching”
via WhatsApp
The COVID-19 pivot was an important shift and Khulisa needed
to triangulate the data

Questions?

Emerging through…
Approaches and
Reflections

Pivoted Planned
VS
Two potential approaches to analysing
WhatsApp Data

23
Pivoted
• Projects pivot due to uncertainty
• WhatsApp is phased in
• Research/evaluation objectives did not
factor the data source in. e.g. pre
COVID-19
• Analytical/evaluation questions for the
data source are formulated on
inspection
First
approach . . .

24
Planned
• Project is planned with tools in mind
• WhatsApp is incorporated from the
onset
• Research Objectives account for the
data source
• Questions are formulated in advance
Second
approach . . .

...Discuss more in the “What if ” section…
7/9/2020 25

Data Collection???
26

Data Collection
27
October 2020
December 8-18 2020
Pivoted

Extraction process
Coaches were requested to extract
WhatsApp media from their phones
and send them to two Khulisa
representatives

ChatBox!
Have you ever “exported chats”
from WhatsApp? Let us know in
the ChatBox!
If not, we’ll show you how to
shortly!

And the
answer is…
for Apple

And the
answer is…
for Android

Challenges
32
Coaches
9
Head Coach
1
Total =14 Total =2
Total =??
Pivoted

Reasons why coaches couldn’t
provide WhatsApp Data
• Some coaches had already returned their
phone to the project implementer so they
couldn't access the chats anymore
• Other coaches had changed phones and
lost their history
• Another coach broke her phone

When we receive the data. . .
●
34

When you save the data. . .
●
35

When you open the t.x.t file
36

Questions?

Analysis…
38

Analysis….
QUALITATIVE: THE QUALITY OF COACHING DURING LOCK DOWN
1. Date of Commencement? End date?
2. What are the conversations in the group? Dominant themes?
3. What kind of multi-media was shared?
4. Is there an influencer/champion?
5. What is the role of the coach, describe the coaching style?
39

6. How do the teachers respond to the group?
7. What is the purpose of the group? Does the group adhere to its objectives e.g.
Fidelity to material; coach communicates with teachers etc.?
8. How does the group respond to COVID-19 challenges?
9. Any other unique information?
40
Analysis….

In practice?
41
Going through EVERYTHING . . .
Reading every line, keeping up with
the threads, looking out for any and
all information in the group, analysing
the shared files in chronology…
Disadvantages of pivoting…

Sleuthing for…
• Time stamps
• Thematic, content and text analysis on the txt files e.g. recurring
words or topics, discussions related to the research; unique topics
e.g. COVID1-19; (Software? )
• Tonality: Formal v Informal
• Mediums shared: listening to voice notes, looking at the shared
pictures and document; chronology?
• How participants respond : interactive, short, vibrant
• How the admin/coach responds: facilitate and encourage discussions;
the culture they create within the group; conflict management.
42

Example 1: From t.x.t files . . .
●
43

…to shared mediums
44

And then what…
45

Life happened in these groups. . .
46
COVID-19
SUPPORT

File extensions. . . <media omitted > and <file attached>
47
• Emojis
• .vcf eg. Mosa.vcf ; 4contacts.vcf= Contacts
• PTT…opus e.g. PTT.20200609-WA000Opus = Voice notes
• IMG…jpg eg IMG0000.jpg = Images ; videos?
• STK-….webp e.g. STK 20200604WA000.Webp = Web files
• Doc.x and .pdf files
Evaluators tip:

Shared Mediums example
48

Questions?

Lastly…. Quantitative?
50

But then….
51

QUANTITATIVE ANALYSIS
7/9/2020 52

QUANTITATIVE ANALYSIS
7/9/2020 53
The analysis considered quantitative questions such as:
• The number of participants in the group?
• Influencers or champions?
• The frequency of the administrator coach’s interaction in the group?
• The frequency of the influencers?

QUANTIFYING WHATSAPP
7/9/2020 54
1. Convert txt file into a structured data set
• Convert txt file into a structured data set that can be used for quantitative analysis
using Python, R or other stats package
• The txt file will be read as 2 columns:
• First column will contain the date
• Second column will contain the time, authors/senders and the message.
• Label the columns accordingly
• Split the second column into time, sender and message

Converting to a structured database

56
Converting to a structured database

7/9/2020 57
2. Clean the dataset
• Remove instances where a message reflects the group creation,
someone was added to the group, some left the group and message
was deleted
• Add new columns from the date column such as day of the week,
month, year, emoji, etc.
• Anonymize data

7/9/2020 58
2. Clean the dataset: Standardize the text (cont.)
• Convert to lower case
• Take out punctuation
• Remove stop words
• Use stemming and lemmatization
Stemming and lemmatization are methods used by
analysts to analyze the meaning behind a word.
Stemming uses the stem of the word while lemmatization,
the context in which the word is being used

7/9/2020 59
3. Quantitative analysis
• Number of messages sent by person (Week, Month, Year)
• Average number of words per messages
• Number of messages per day
• Wordcloud

Questions?

Group
Sizes
Group
size
Number of
Groups
<10 2
10-20 6
21-30 3
>30 3
Total Participants
36
5
Groups
1
4
Group Participants
Coach 1 92
Coach 9 71
Coach 3 35
Coach 17 29
Coach 10: Grade 1 22
Coach 11 21
Coach 15 (Head coach) 17
Coach 16: Primary School
1
16
Coach 13 15
2
12
3
8
Coach 7 3
4,693 Messages exchanged

Group
Sizes
Group
size
Number of
Groups
<10 2
10-20 6
21-30 3
>30 3
Total Participants
365
Groups
14
4,693
Messages
exchanged
Group Participants
Coach 1 92
Coach 9 71
Coach 3 35
Coach 17 29
Coach 11 21
Coach 15 (Head coach) 17
Coach 16: Primary School 1 16
Coach 13 15
Coach 7 3
Results

Groups
Coach
messages*
Influencer
messages**
Total group
messages
Coach
messages
%
Coach 1 552 302 2147 26%
Coach 9 353 137 1390 25%
Coach 3 92 12 256 36%
Coach 17 25 9 105 24%
Coach 10 Grade 1 54 17 149 36%
Coach 11 15 5 59 25%
Coach 15 (Head Coach) 53 13 161 33%
Coach 16:
Primary School 1
16 4 40 40%
Coach 13 24 17 111 22%
Coach 10 Grade 3 30 5 55 55%
Coach 16:
Primary School 2
14 4 34 41%
Coach 10 Grade 2 35 24 88 40%
Coach 16:
Primary School 3
15 11 39 38%
Coach 7 41 16 59 69%
* “Coach messages” refer to the number of
times the coach sent out messages
** “Influencer messages” refer to the
number of contributions from the top
member in the group

COVID-19 Pivot - Findings
6/2/2021 67
Coaches reported that they would call teachers to try and offer
more support and guidance, but it was difficult to coach over the
phone
Coaches also reported that teachers were often unresponsive
Reportedly the cost of data became an issue:
“The interaction was there, but not that much. Mobile data
is a challenge. Some teachers said they didn't download
what I sent them because they don't have enough data.”
(Coach 9)

Questions?

Planned approach…
What if?

70
• Project is planned with tools in mind
• WhatsApp is incorporated from the onset
• Evaluator is a co-administrator on the WhatsApp
groups eliminating the need to ask others to
download data
• Research objectives account for the data source
• Questions are formulated before hand
Planned

What to consider when planning. . .
7/9/2020
71
1. Evaluation/Program Objectives
Questions:
• Will it assist our program objectives?
• Usage and technical support, are participants
techno savvy and supported?
• Administration (who, how), rules of
engagement for the community ?
• The granular, the better

7/9/2020
72
2. Legal and Ethical Considerations
• Protection of personal information
policies in your country. Speak to
participants beforehand and get
consent
• Make sure ethical clearance accounts for
the source before hand
• Revoking consent?

Strike through…
●
73

7/9/2020
74
3. Data collection planning and execution:
• Overall easier process
• Administrators
• Evaluators

7/9/2020
75
4.Analysis
Software for thematic, content and text analysis

Questions?

Evaluator Checklist
• Research Design Analysis: Is the use of WhatsApp aligned
to your objectives? Technical support? Community rules?
• Protect personal information and consent
• Identify administrator (co-administrator could be evaluator)
• Download chat timeline – how often
• Conduct analysis
• Designate Results
77

Lessons learned for Projects/Programs
78
• Just telling colleagues to set up WhatsApp groups with their
beneficiaries is insufficient
• Provide guidance on rules of engagement
• Ensure consent and privacy is respected
• Decide if participants can be given data to post and receive resources
OR resources can be loaded on a zero-rated website
• Share resources/posts
• Identify influencers to engage further
• If an important intervention, ensure that you set up monitoring system
• Incorporate results into your data-driven decision making

Key Reports
• Design / Inception Report (2019)
• Formative Evaluation Report (2020)
• Summative Evaluation Report (2021)
All evaluation reports available on
https://dec.usaid.gov/dec/home/Default.aspx

Questions?
Comments?

THANK YOU !
7/9/2020 81
DISCLAIMER
THE AUTHOR’S VIEWS EXPRESSED IN THIS PRESENTATION DO NOT NECESSARILY REFLECT THE VIEWS OF THE UNITED STATES
AGENCY FOR INTERNATIONAL DEVELOPMENT OR THE UNITED STATES GOVERNMENT OR DTI OR LASER PULSE

gLOCAL 2021 WhatsApp Analysis webinar presentation KhulisaMS

Recommended

Recommended

More Related Content

Similar to gLOCAL 2021 WhatsApp Analysis webinar presentation KhulisaMS

Similar to gLOCAL 2021 WhatsApp Analysis webinar presentation KhulisaMS (20)

More from Khulisa Management Services

More from Khulisa Management Services (20)

Recently uploaded

Recently uploaded (20)

gLOCAL 2021 WhatsApp Analysis webinar presentation KhulisaMS