1. Frontiers of
Computational Journalism
Columbia Journalism School
Week 3: Information Filter Design
September 26, 2016

2. This class
• The need for information filtering
• Filtering algorithms
• Human-machine filters
• Filter bubbles and other problems
• The filter design problem

3. The Need for Filtering

7. More video on YouTube than produced by TV networks during
entire 20th century.

8. 10,000 legally-required reports filed by U.S. public
companies every day

9. Each day, the Associated Press publishes:
~10,000 text stories
~3,000 photographs
~500 videos
+ radio, interactive…

10. Comment Ranking

11. Comment voting
Problem: putting comments with most votes at top doesn’t work.
Why?

12. Old reddit comment ranking
“Hot” algorithm.
Up – down votes plus time
decay

13. Reddit Comment Ranking (new)
Hypothetically, suppose all users voted on the comment, and v out of N
up-voted. Then we could sort by proportion p = v/N of upvotes.
N=16
v = 11
p = 11/16 = 0.6875

14. Reddit Comment Ranking
Actually, only n users out of N vote, giving an observed approximate
proportion p’ = v’/n
n=3
v’ = 1
p’ = 1/3 = 0.333

15. Reddit Comment Ranking
Limited sampling can rank votes wrong when we don’t have enough
data.
p’ = 0.333
p = 0.6875
p’ = 0.75
p = 0.1875

16. Confidence interval
1-𝛼 probability that the true value p will lie within the central
region (when sampled assuming p=p’)

17. Rank comments by lower bound
of confidence interval
p’ = observed proportion of upvotes
n = how many people voted
zα= how certain do we want to be before we assume that p’ is “close” to
true p
Analytic solution for confidence interval, known as “Wilson score”
How not to sort by average rating, Evan Miller

18. User-item Recommendation

19. User-item matrix
Stores “rating” of each user for each item. Could also be
binary variable that says whether user clicked, liked,
starred, shared, purchased...

20. User-item matrix
• No content analysis. We know nothing about what is “in” each item.
• Typically very sparse – a user hasn’t watched even 1% of all
movies.
• Filtering problem is guessing “unknown” entry in matrix. High
guessed values are things user would want to see.

21. Filtering process
Item-Based Collaborative Filtering Recommendation Algorithms, Sarwar et al

22. How to guess unknown rating?
Basic idea: suggest “similar” items.
Similar items are rated in a similar way by many different users.
Remember, “rating” could be a click, a like, a purchase.
o “Users who bought A also bought B...”
o “Users who clicked A also clicked B...”
o “Users who shared A also shared B...”

23. Similar items
Item-Based Collaborative Filtering Recommendation Algorithms, Sarwar et al

24. Item similarity
Cosine similarity!

25. Other distance measures
“adjusted cosine similarity”
Subtracts average rating for each user, to compensate for general
enthusiasm (“most movies suck” vs. “most movies are great”)

26. Generating a recommendation
Weighted average of item ratings by their similarity.

27. Matrix factorization recommender

28. Matrix factorization recommender
Note: only sum over observed ratings rij.

29. Matrix factorization plate model
r
v
u
user rating
of item
variation in
user topics
λu
λv
variation in
item topics
topics for user
topics for item
i users
j items

30. New York Times recommender

31. Different Filtering Systems
Content:
Newsblaster analyzes the topics in the documents.
No concept of users.
Social:
What I see on Twitter determined by who I follow.
Reddit comments filtered by votes as input.
Amazon "people who bought X also bought Y” - no content analysis.
Hybrid:
Recommend based both on content and user behavior.

32. Combining collaborative filtering
and topic modeling
Collaborative Topic Modeling for Recommending Scientific Articles, Wang and Blei

33. K topics
topic for word word in doc
topics in doc
topic
concentration
parameter
word
concentration
parameter
Content modeling - LDA
D docs
words in topics
N words
in doc

34. K topicstopic for word word in doctopics in doc
(content)
topic
concentration
weight of user
selections
variation in
per-user topics
topics for user
user rating
of doctopics in doc
(collaborative)
Collaborative Topic Modeling

35. content only
content +
social

36. Filtering News on Twitter

40. Reuters News Tracer
Filter
Cluster into
events
Searches
and Alerts
Score
veracity &
newsworthy

41. Liu et. al, Reuters Tracer: A Large Scale System of Detecting &
Verifying Real-Time News Events from Twitter

42. Liu et. al, Reuters Tracer: A Large Scale System of Detecting &
Verifying Real-Time News Events from Twitter

43. Liu et. al, Reuters Tracer: A Large Scale System of Detecting &
Verifying Real-Time News Events from Twitter

44. Human-Machine Filters

45. TechMeme / MediaGazer

46. Facebook trending (with editors)

47. Facebook trending (without editors)

48. Facebook “trending review tool” screenshot from leaked documents

49. Approve or Reject: Can You Moderate Five New York Times Comments?

50. Revealed: Facebook's internal rulebook on sex, terrorism and violence, The Guardian

51. Facebook’s “Community Standards” document

52. Filter bubbles and other problems

53. Graph of political book sales during 2008 U.S. election, by orgnet.org
From Amazon "users who bought X also bought Y" data.

54. Retweet network of political tweets.
Political Polarization on Twitter, Conover, et. al.,

55. Instagram co-tag graph, highlighting three distinct topical communities: 1) pro-Israeli
(Orange), 2) pro-Palestinian (Yellow), and 3) Religious / muslim (Purple)
Gilad Lotan, Betaworks

56. The Filter Bubble
What people care about politically, and what they’re motivated to do something
about, is a function of what they know about and what they see in their media.
... People see something about the deficit on the news, and they say, ‘Oh, the
deficit is the big problem.’ If they see something about the environment, they
say the environment is a big problem.
This creates this kind of a feedback loop in which your media influences your
preferences and your choices; your choices influence your media; and you
really can go down a long and narrow path, rather than actually seeing the
whole set of issues in front of us.
- Eli Pariser,
How do we recreate a front-page ethos for a digital world?

57. Are filters causing our bubbles?
Increasing U.S. polarization predates Internet by decades.

58. Is the Internet Causing Political Polarization? Evidence from Demographics
Boxell, Gentzkow, Shapiro
Polarization increasing fastest
among those who are online the least

59. Exposure to Diverse Information on Facebook,
Eytan Bakshy, Lada Adamic, Solomon Messing
Will you see diverse content vs. will you click it?

60. Filter Design

61. Item Content My Data Other Users’ Data
Text analysis,
topic modeling,
clustering...
who I follow
what I’ve read/liked
social network
structure,
other users’ likes

62. Filter design problem
Formally, given
U = user preferences, history, characteristics
S = current story
{P} = results of function on previous stories
{B} = background world knowledge (other users?)
Define
r(S,U,{P},{B}) in [0...1]
relevance of story S to user U

63. Filter design problem, restated
When should a user see a story?
Aspects to this question:
normative
personal: what I want
societal: emergent group effects
UI
how do I tell the computer I want?
technical
constrained by algorithmic possibility
economic
cheap enough to deploy widely

64. “Conversational health”
Measuring the health of our public conversations, Cortico.ai

65. Exposure diversity as a design principle for recommender systems, Natali Helberger

66. How to evaluate/optimize?

67. How to evaluate/optimize?
• Netflix: try to predict the rating that the user gives a movie
after watching it.
• Amazon: sell more stuff.
• Google, Facebook: human raters A/B test every change (but
what do they optimize for?)

68. • Does the user understand how the filter works?
• Can they configure it as desired?
• Controls for abuse and harassment
• Can it be gamed? Spam, "user-generated censorship," etc.
How to evaluate/optimize?

69. Information diet
The holy grail in this model, as far as I’m
concerned, would be a Firefox plugin that would
passively watch your websurfing behavior and
characterize your personal information
consumption. Over the course of a week, it might
let you know that you hadn’t encountered any
news about Latin America, or remind you that a full
40% of the pages you read had to do with Sarah
Palin. It wouldn’t necessarily prescribe changes in
your behavior, simply help you monitor your own
consumption in the hopes that you might make
changes.
- Ethan Zuckerman,
Playing the Internet with PMOG