CS Seminar talk by Dr Kokil Jaidka 15 April 2019 Video Recording of the talk here: https://youtu.be/gKWatqnR2fI

1. Quasi-Experimental Methods for
Social Media Analysis
Dr. Kokil Jaidka
Presidential Postdoctoral fellow ‘18

2. Research Statement
Advancing theories about communication
using computational linguistics

3. Research Statement
Advancing theories about communication
using computational linguistics

4. Language is not about what words mean;
language is about what people mean.
- Herbert Clark

5. Feature Implication
Modeling personality/conversation traits
from social media posts
• Guntuku, S., Buffone, A., Jaidka, K., Eichstaedt, J., Ungar L. (Forthcoming). Understanding and
Measuring Psychological Stress using Social Media. In Proceedings of the 13th International
Conference on Web and Social Media (ICWSM 2019). AAAI.
• Rouhizadeh, M., Jaidka, K., Smith, L., Buffone, A., Schwartz, H. A., Ungar, L. (2018). Identifying
Locus of Control in Social Media Language. In Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing (EMNLP 2018). ACL.
• Jaidka, K., Chhaya, N., Ungar, L. (2018). Diachronic degradation of language models: Insights
from social media. In Proceedings of the 56th Annual meeting for the Association for
Computational Linguistics (ACL 2018). ACL.
The Big Picture

6. status update!
status update!
status update!
status update!
status update!
tweet!
tweet!
tweet!
tweet!
tweet!
Slide courtesy H. Andrew Schwartz, 2017

7. status update!
status update!
status update!
status update!
status update!
tweet!
tweet!
tweet!
tweet!
tweet!
Outcomes
Slide courtesy H. Andrew Schwartz, 2017

8. tweet!
tweet!
tweet!
tweet!
tweet!
status update!
status update!
status update!
status update!
status update!
language
analysis
Outcomes
Slide courtesy H. Andrew Schwartz, 2017

9. tweet!
tweet!
tweet!
tweet!
tweet!
status update!
status update!
status update!
status update!
status update!
language
analysis
● prediction
(measurement)
● insights
Outcomes
Slide courtesy H. Andrew Schwartz, 2017

10. extraversion -- sociable, assertive, active, energetic, talkative, outgoing
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M.
E. P., & Ungar, L. H. (2013). Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach. In PLOS
Slide courtesy H. Andrew Schwartz, 2017

11. introversion
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D.,
Seligman, M. E. P., & Ungar, L. H. (2013). Personality, Gender, and Age in the Language of Social Media: The
Open-Vocabulary Approach. In PLOS ONE 8(9). Slide courtesy H. Andrew Schwartz, 2017

12. Gende
r
Slide courtesy H. Andrew Schwartz, 2017

13. Feature Implication
Modeling personality/conversation traits
from social media posts
• Guntuku, S., Buffone, A., Jaidka, K., Eichstaedt, J., Ungar L. (Forthcoming). Understanding and
Measuring Psychological Stress using Social Media. In Proceedings of the 13th International
Conference on Web and Social Media (ICWSM 2019). AAAI.
• Rouhizadeh, M., Jaidka, K., Smith, L., Buffone, A., Schwartz, H. A., Ungar, L. (2018). Identifying
Locus of Control in Social Media Language. In Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing (EMNLP 2018). ACL.
• Jaidka, K., Chhaya, N., Ungar, L. (2018). Diachronic degradation of language models: Insights
from social media. In Proceedings of the 56th Annual meeting for the Association for
Computational Linguistics (ACL 2018). ACL.
Testing the theories about political
communication (campaigning, opinion
expression, deliberation)
• Jaidka, K., Zhou, A., Lelkes, Y. (Revise and resubmit). Brevity is the soul of Twitter: The
constraint affordance and political discussion. Journal of Communication.
• Jaidka, K., Ahmed, S., Skoric, M., Hilbert, M. (2018). Predicting Elections from Social Media: A
Three-Country, Three-Method Comparative Study. Asian Journal of Communication (Online
first), 1-21.
• Ahmed, S., Jaidka, K., Cho, J. (2018). Do birds of a different feather flock together?
• Analyzing the political use of social media through a language-based approach in a multi-lingual
context. Computers in Human Behavior, 86, 299-310.
• Ahmed, S., Cho, J, Jaidka, K. (2018). Framing social conflicts in news coverage and social media:
a multi-country comparative study. International Communication Gazette.
The Big Picture

14. Slide courtesy Yphtach Lelkes, 2019

15. Slide courtesy Yphtach Lelkes, 2019

16. Slide courtesy Yphtach Lelkes, 2019

17. More
Same
Less
Respectful Civil
Thinking about all of the places OUTSIDE OF SOCIAL MEDIA where
people might discuss politics or political issues, are the political
discussions you see on social media...
Informative Angry
Pew's American Trends Panel W19,
2016
Slide courtesy Yphtach Lelkes, 2019

18. More
Same
Less
Respectful Civil
Thinking about all of the places OUTSIDE OF SOCIAL MEDIA where
people might discuss politics or political issues, are the political
discussions you see on social media...
Informative Angry
Pew's American Trends Panel W19,
2016
Slide courtesy Yphtach Lelkes, 2019

19. Make the (online) world a better place
● Censor social media accounts (King, Pan, & Roberts, 2014)
● Issue interventions (Munger, 2016; Stroud, Duyn, & Peacock, 2016)
● Design better platforms (Affordances framework)

20. TODAY’S
RESEARCH QUESTION
20
Longer posts “cause”
richer self-expression
Longer posts are related
to richer self-expression
The design of social media platforms affects
the health of online conversations

21. Prior work
● Differences in message-length affordances are related to differences
in the disinhibition effect
○ Youtube comments are more toxic than news website comments (Halpern &
Gibbs, 2013; Theocharis et al., 2015; Towne & Herbsleb, 2012)
○ Longer messages in political discussions were significantly more uncivil than
shorter messages (Papacharissi, 2004).
○ Shorter messages on Twitter were significantly less deliberative than longer
messages on Facebook (Oz, Zheng, and Chen, 2018).
21
The design of social media platforms affects
the health of online conversations

22. ● [RQ1] What effect did doubling the character limit on Twitter have on
the characteristics of political discussions?
● [RQ2] What are the effects for different bandwidths (time periods)?
● [RQ3] What are the average treatment effects among the compliers?
Self-selection check: What are the effects controlling for individual differences?
Robustness check: What are the effects controlling for the length of tweets?
Placebo test: What were the effects on the same day, a year ago?
22
Research Questions
The design of social media platforms affects
the health of online conversations

23. 23
Dependent Variables
Incivility
•Offensiveness
•Uncivil words
•Swearing
•Politeness
Deliberation
•Analytical thinking
•Cognitive processing
Relevance
•Political words
•Political entities
•Media entities
• “‘@ DO YOU WANT TO BE REMEMBERED AS
A GIANT Pxxxx. U GERRYMANDERED
DISTRICTS 4 LYFE. U DUMB MOTHER Fxxxxx
NEED TO SLOW THE ROLL.”
• 2.64 percent of sample
• “@ Impeach the Whitehouse Hitler.
CRIMINAL CRIMINAL …”
• 1.43 percent of our sample
Dataset: 350,000 Twitter replies to US members of the Congress between Jan 2017 and March 2018
filtered from the Twitter random 1% sample. Twitter changed the character limit on Nov 7, 2017.
The design of social media platforms affects
the health of online conversations

24. 24
Dependent Variables
Incivility
•Offensiveness
•Uncivil words
•Swearing
•Politeness
Deliberation
•Analytical thinking
•Cognitive processing
Relevance
•Political words
•Political entities
•Media entities
• “@ Why don’t you Republicans stop praying
and do something! You’re just prostitutes
for the NRA.”; .08 in politeness.”
• “@ Please consider my attached letter. I
sent it to your colleague, Sen. as well. I
appreciate your service…thank you!” .92 in
politeness.
Dataset: 350,000 Twitter replies to US members of the Congress between Jan 2017 and March 2018
filtered from the Twitter random 1% sample. Twitter changed the character limit on Nov 7, 2017.
The design of social media platforms affects
the health of online conversations

25. 25
Dependent Variables
Incivility
•Offensiveness
•Uncivil words
•Swearing
•Politeness
Thinking style
•Analytical thinking
•Cognitive processing
Relevance
•Political words
•Political entities
•Media entities
Dataset: 350,000 Twitter replies to US members of the Congress between Jan 2017 and March 2018
filtered from the Twitter random 1% sample. Twitter changed the character limit on Nov 7, 2017.
• “@ Anti-immigrant is un-American.
Oppose racism/sexism/hate.
OPPOSE Sessions for AG! Demand
better for America! #StopSessions.”
• “@USER @USER Umm, no it isn’t.”
• “@ Can’t make any promises but I’ll
try.”
• ““@USER Wake up and be a GOP
Senator!!!!”
The design of social media platforms affects
the health of online conversations

26. 26
Dependent Variables
Incivility
•Offensiveness
•Uncivil words
•Swearing
•Politeness
Thinking style
•Analytical thinking
•Cognitive processing
Relevance
•Political words
•Political entities
•Media entities
Dataset: 350,000 Twitter replies to US members of the Congress between Jan 2017 and March 2018
filtered from the Twitter random 1% sample. Twitter changed the character limit on Nov 7, 2017.
The design of social media platforms affects
the health of online conversations

27. 27
Interrupted Time Series
[RQ1] What effect did doubling the character limit on Twitter
have on the characteristics of political discussions?
• Aggregate tweets to the day level
• Bandwidth is 100 days before and after switchover
• T is the relative time distance from November 7. 2017
• X is a dummy variable indicating whether the tweets were published
before (coded 0) or after intervention (coded 1)
The design of social media platforms affects
the health of online conversations

28. 28
Fuzzy Regression Discontinuity Design
[RQ3] What are the Local Average Treatment Effects among
the compliers?
http://web.mit.edu/teppei/www/teaching/Keio2016/05rd.pdf
• Estimates the effect among those who actually complied with
treatment (LATE).
• 2SLS approach where running variable is the instrumental
variable and binary >145 characters endogenous variable
The design of social media platforms affects
the health of online conversations

29. 29
Fixed effects
Self-selection check.
+ 𝜶𝒊
The design of social media platforms affects
the health of online conversations

30. 33
Results
ITS: .02, ns
within-subject: .03, ns
IV Estimate: .08, ns
ITS: -.09***
within-subject: -.10***
IV Estimate: -.42***
ITS: -.02
within-subject: -.004.
IV Estimate: -.12**
ITS: .07***
within-subject: .02, ns
IV Estimate: .31***
[RQ1,RQ3] What effect did doubling the character limit on
Twitter have on the characteristics of political discussions?
The design of social media platforms affects
the health of online conversations
With 140 extra characters, political tweets on Twitter
are more civil (3/4 operationalizations)

31. 34
Results
ITS: .02, ns
within-subject: .03, ns
IV Estimate: .08, ns
ITS: -.09***
within-subject: -.10***
IV Estimate: -.42***
ITS: -.02
within-subject: -.004.
IV Estimate: -.12**
ITS: .07***
within-subject: .02, ns
IV Estimate: .31***
[RQ1,RQ3] What effect did doubling the character limit on
Twitter have on the characteristics of political discussions?
The design of social media platforms affects
the health of online conversations
With 140 extra characters, political tweets on Twitter
are more civil (3/4 operationalizations)

32. 35
Results
ITS: .02, ns
within-subject: .03, ns
IV Estimate: .08, ns
ITS: -.09***
within-subject: -.10***
IV Estimate: -.42***
ITS: -.02
within-subject: -.004.
IV Estimate: -.12**
ITS: .07***
within-subject: .02, ns
IV Estimate: .31***
[RQ1,RQ3] What effect did doubling the character limit on
Twitter have on the characteristics of political discussions?
The design of social media platforms affects
the health of online conversations

33. 37
Results
ITS: .04
within-subject: .06
IV Estimate: .06
ITS: -.03
within-subject:-.02
IV Estimate: -.07
[RQ1,RQ3] What effect did doubling the character limit on
Twitter have on the characteristics of political discussions?
The design of social media platforms affects
the health of online conversations
Those who use the 140 extra characters in their political tweets
show more analytical thinking and cognitive processing.

34. 39
Results
ITS: -.07***
within-subject: -.05***
IV Estimate: -.40***
[RQ1,RQ3] What effect did doubling the character limit on
Twitter have on the characteristics of political discussions?
The design of social media platforms affects
the health of online conversations
With 140 extra characters, tweets became less relevant to politics.

35. 40
Results: Time period
[RQ2] What are the effects for different bandwidths?
The design of social media platforms affects
the health of online conversations
Findings are stable across
different choices of time period

36. 43
Results: Time period
[RQ2] What are the effects for different bandwidths?
The design of social media platforms affects
the health of online conversations

37. 44
Recap of Results
With 140 extra characters
• Tweets are more civil
• Tweets show more analytical thinking and cognitive processing,
i.e., higher rationality.
• Tweets are less relevant to politics, i.e., less substantive.
Findings are stable across different time periods, accounting for
self-selection bias, beyond placebo effects and limitations of how
the DV was operationalized.
The design of social media platforms affects
the health of online conversations

38. Theoretical
Implications
• Less substantive discussions imply
the absence of “actual” deliberation,
or consensus-building
• Changing social norms imply
changing political interests and
political engagement
• Stricter social norms can marginalize
counter-narratives and create echo
chambers

39. Limitations
46

40. Surveys Experiments Observations
Future work
The interplay of affordances
in democratizing online
spaces
The role of anonymity
affordances in disinhibiting
self-expression and criticism
amongst Singaporeans
47
The role of privacy
affordances in buffering
well-being and self-
efficacy: a cross-national
comparison

41. Ethical concerns
● 2014: Lax Facebook customer data protection means that Cambridge Analytica gains access to 50 million
user profiles (USA)
● 2016: Insurance company to use Facebook profile to determine insurance premium (UK)
● 2017: Courts are now using AI to sentence criminals (USA)
● 2017: Homeland Security monitors immigrants’ social media profiles (USA)
● 2018: Airline passengers to face AI lie detector tests in airports (EU)
"This is part of a broader trend towards using opaque, and often deficient,
automated systems to judge, assess and classify people" : Frederike Kaltheuner,
Privacy International
48

42. Thank you
Q & A