This document discusses building scalable and personalized news feeds. It defines key concepts like actors, verbs, objects, actions, fan outs, followers, and items. Popular feeds from Instagram, Twitter and Facebook are examined. The main challenges of aggregating posts from followed users into a unified display are explained. Guidelines are provided on replicating data, using simple queries, and scoring relevancy. Terminology around feeds following other feeds is introduced. Finally, considerations for modifying fanned-out items and using existing pub/sub mechanisms are discussed.

1. Building Scalable and
Personalized News Feed
“activity streams”
Eyal Ronel for Web Developers Community
https://www.meetup.com/Web-Developers-Community

2. Eyal Ronel
R&D Manager at
https://www.linkedin.com/in/eyalronel
https://github.com/EyalRonel
@eyalronel
http://www.slidernet.co

3. Overview
1. Getting familiar with the concepts of activity streams
2. Differentiating between feeds and “dashboards”
3. Understanding the challenges
4. Guidelines on how to build you own feed mechanism

4. News Feed
What is it?

6. Not A Feed…
Bank Leumi - iOS App

7. Not A Feed…
CNN.com latest Headlines Officialcharts.com top rated songs

8. Popular Feeds

9. Instagram
• Users can follow/unfollow other users
• A News feed (used to) consists of all
items submitted by users you’re
following
• Feed Items include images, videos and
ads
• Sorted chronologically

10. Twitter
• Users can follow/unfollow other users
• A News feed consists of all items
submitted by users you’re following
• Feed Items include textual tweets,
images, videos and ads
• Sorted chronologically

11. Facebook
• Users can friend/unfriend other users
• User can follow/unfollow other users
(new friends are followed by default)
• A News feed consists of selected
items submitted by users you’re
following (not only your friends)
• Feed Items include status updates,
images, videos, checkins, ads, etc
• Sorted by Relevancy(!)

12. +450
+5+3
Understanding the challenge

13. Understanding the challenge
+450
+5+3

14. Understanding the challenge

15. The challenge
• Main challenge (back end):
Aggregating all posts, from all “followed” users ,
into a single, unified display
• Paging a constantly updating list of items
• Displaying newly added feed items after initial
rendering

16. Can this work?
?

17. Lets try..
SELECT * FROM POSTS
WHERE user_id IN (<followerID1>,…<FollowerIdN>)
LIMIT 10
OFFSET 0
ORDERBY date
What if I’m following 1,000 users?
10,000 users? 100K users?
SELECT * FROM Following
WHERE user_id = “<My User ID>
LEFT JOIN Users
ON users.id = Following.user_id
Far from optimal, does not scale

18. Conclusions
• A Complicated / Resource consuming query is
more suitable for reports / dashboards
• A Feed is not a Dashboard

19. Data - Items share the same data
structure, usually from a single data
source
Visual - Items are visually similar
Sorting - Items are usually presented in
an ordered list (by “Score” or Date)
A result of an “aggregation” query
(JOIN, GROUP, etc)
• Data - Items are “enveloped”,
and may contain different data
structures, from multiple sources
• Visual - Different Visual for
different feed items, depending
on their type
• Sorting - “Relevancy scoring”
• A result of a simple query!
…but how?
Dashboard Feed

20. NoSQL
• Replicate data
• Work hard on insert
• Use simple queries to fetch

21. Feed Terminology
Actor
Verb
Object ID
Fan out Feed ItemsAction Followers
Verb Feed
1
Relevancy
Scoring!

22. Feed Terminology
Actor - A user performing an action
(status update, image post, other)
Verb - Unique value describing the action’s
type.
for example: 1 = status update, 2 = image
Action - A record containing the actor’s id,
verb, object id and additional meta-data.
An action will be eventually become
multiple feed items

23. Feed Terminology
Fan out - The process of distributing a
single action to all followers. turning an
action to multiple items.
Followers - A list of users subscribed to
updates for a specific feed
Object ID - The object ID (database id) of
the entity the action is describing
* (A post about an image, is not the image itself)

24. Feed Terminology
Item - A single instance of, or reference to
an action inside a user’s feed
Feed - A list of items assigned/associated
with a specific actor (user)

25. Feed Data Flow
Action Feed items

26. People don’t follow people,
feeds follow feeds
One last thing before we can start…
In many cases, a single user has 2 feeds:
• News feed - a private feed containing items from his/her friends.
Technically, a user’s news feed is following other public feeds
(profiles/timeline feeds)
• Profile (timeline) - a public feed listing all action performed by the
current user.A ‘profile’ is actually a feed where all items have the same
actor
(the feed’s owner)

27. Feeds follow Feeds
My private
news feed
Yaron’s
public feed
Noa’s
public feed/profile
Itai’s
public feed
Yaron’s
public feed
Eyal’s
public feed/profile
Itai’s
public feed
Noa’s private
news feed

28. Feed Actions
• Modifying fanned-out feed items editing a “fanned-
out” post
• Adding Likes
• Adding Shares
• Existing Pub/Sub mechanisms
• Using relational databases - why/why not?

29. DEMO