1) The document summarizes a presentation on using social media data to measure gender norms. It outlines studies analyzing tweets about age-discordant relationships in 10 DREAMS countries to track attitudes toward gender norms and gender-based violence. 2) Preliminary results found 97% of tweets were from South Africa and sentiment analysis of 1,766 tweets showed computer coding matched human coding 41% of the time. 3) Both opportunities and challenges of using social media to evaluate gender norms are discussed, including ability to collect attitudes but also risks of selection bias and challenges accurately analyzing nuances.

1. Big Data Analytics for
Measuring Gender
Norms: Too Big to Ignore
Carolina Mejia, PhD, MPH (presenter)
Brittany Schriver Iskarpatyoti, MPH
John Spencer, MA
Jessica Fehringer, PhD, MPH
Paul Brodish, PhD
MEASURE Evaluation
University of North Carolina
November 8, 2017
AEA Conference, Washington, D.C.

2. Session Objectives
• Review current advances in online social media
data for measuring public attitudes and behaviors
• Illustrate examples of how social media data
sources can be used to analyze gender norms
• Briefly present the methodological opportunities
and limitations of these approaches to evaluation

3. Background
• Gender norms and power dynamics affect people’s
health and sexual behavior.
• There is a dearth of gender norms data that are
readily accessible, collected, or used; this may
leave critical information gaps.
• Social media provide opportunities to fill these
gaps.
• However, the suitability of these techniques for
assessing gender norms is uncertain.

4. Study Objectives
• Explore the feasibility of using large social media
datasets to track changes in attitudes toward
gender norms on sexual relationships between
younger woman and older men and gender-
based violence.
• Share findings that will guide recommendations
for using such data to answer PEPFAR gender-
related evaluation questions and more efficiently
dedicate resources.

5. Methods
• Source: Twitter
• Searched tweets on the topic of age-
discordant relationships (i.e., blessers, sugar
daddy, etc.) using Crimson Hexagon, a
social media analytics platform.
• Pulled retrospective data (1/1/15 to 1/31/17)
in 10 *DREAMS countries.
• Excluded re-tweets.
• n= 10,000
• Extracted approximately 20 percent of
tweets (n=1766) that had indicated the sex
of the user (male/female) for qualitative
analysis.
* DREAMS -- Determined, Resilient, Empowered, AIDS-free, Mentored, and Safe women
10 DREAMS Countries:
Kenya, Lesotho, Malawi,
Mozambique, South
Africa, Swaziland,
Tanzania, Uganda,
Zambia, and Zimbabwe

6. Preliminary Results
Quantitative Analysis
(n=10,000)

7. Preliminary Results
97% of tweets
were from
South Africa

8. Preliminary Results (2)
Sentiment Analysis
Crimson Hexagon
categorized tweets
as:
• “Basic Negative”
• “Basic Neutral”
• “Basic Positive”
Also, assigned
emotions to each
tweet.

9. Preliminary Results (3)
Klout Score

10. Preliminary Results
Qualitative Analysis
(n=1776)

11. Preliminary Results
Sentiment Analysis
Compared computer-generated with human-coded
sentiment analysis
• 1,777 tweets coded
• 11 tweets removed for lack of relevance
• 1,766 tweets analyzed
Overall, 41% of the codes matched
• Basic Neutral (28%)
• Basic Negative (54%)
• Basic Positive (41%)
• Unknown/Blank (84%)

12. Crimson Hexagon
Positive Negative Neutral
Unknown/
Blank
TOTAL
HumanCoded
Positive 231 52 203 7 493
Negative 108 159 139 4 410
Neutral 139 56 221 11 427
Unknown/
Blank
77 26 221 112 436
TOTAL 563 294 784 134 1766
Sentiment Analysis
Computer vs. Human

13. Sentiment Analysis
Computer vs. Human

14. Discussion

15. Opportunities
• Ability to collect significant amount of data that
reflects people's opinions and comments on society
including gender attitudes and norms
• User-generated, human-centered data
• It’s possible to know (or estimate with reasonable
certainty) the demographic and geographic details
of the user as well as their social connectivity (or
influence)
• Rapid growth in understanding characteristics (sex,
age, etc.) of the users

16. Challenges (1)
• Differences in computer vs. human coding
sentiment analysis
• Gaps in gender identification, especially for
countries where people have unique names that
cannot be deciphered by current computer
algorithms.
• Selection bias: Certain groups are more likely to
use social media than others, depending on
factors such as age, sex, geographic location,
income, and education.

17. Challenges (2)
• You need programming skills if you don’t use
commercial platforms such as Crimson Hexagon.
• Computer algorithms may be challenged in
capturing nuances and sarcasm accurately.
• “Bots” can imitate human behavior thus challenging
our analysis and findings.

18. Ethical and Legal Implications
• The lack of ethical and legal guidance puts social
media users potentially at risk and leaves
researchers with unanswered questions.
• Internal Review Boards (IRBs) are challenged by
making decisions on protecting confidentiality of
public data.
• Laws surrounding issues such as the copyright
ownership of literary works such as blogs, pictures,
videos, and recordings should be carefully
monitored.

19. Conclusions
 Social media data can supplement surveys in
practical ways to reduce costs, increase timeliness,
and expand analytic capacity.
 Social media data can be part of the toolkit to track
changes over time on gender norms and provide a
platform to conduct in-depth analysis of tweets to
inform programs.
 Early indications are that social media may provide
only limited insight into gender norms and likely
require human input, but we'll know more after our
work concludes.

20. Any Questions?
For additional information, contact:
Carolina Mejia
cmejia@unc.edu
John Spencer
john_spencer@unc.edu
Brittany Schriver Iskarpatyoti
bschriver@unc.edu
Jessica Fehringer
fehringe@email.unc.edu

21. This presentation was produced with the support of the United States Agency for
International Development (USAID) under the terms of MEASURE Evaluation
cooperative agreement AID-OAA-L-14-00004. MEASURE Evaluation is
implemented by the Carolina Population Center, University of North Carolina at
Chapel Hill in partnership with ICF International; John Snow, Inc.; Management
Sciences for Health; Palladium; and Tulane University. Views expressed are not
necessarily those of USAID or the United States government.
www.measureevaluation.org