Building ZingMe News Feed System

Building Zingme News Feed
System

Chau Nguyen Nhat Thanh
Senior Software Manager
User Platform Division - VNG

Agenda

1) Introduction to News Feed
2) ZingMe News Feed system history
3) ZingMe News Feed system
4) Some statistics
5) Bonus
6) Q&A

Introduction to News Feed
● Update friends' info
● Update Biz info
● Update VIP info
● Interaction to them by comment, like ...

● Terms
● Social graph: Users in most social networking
sites are describable in terms of a social
graph. The relationships between users are
represented by adjacency lists. If Jack and Jill
are friends, they are said to be adjacent. This
is known as an "edge" in the graph. (from
Quora)
● Not only Friends
● but also Followers …

● What do we need?
● Someone does actions, his friend will see these
action in his home as soon as possible
● What will we solve the problems?
● Solution 1: Push model (fan out on write)
● Solution 2: Pull model (fan out on read)
● Solution 3: Mixing push and pull (Feeding
Frenzy- a paper from Yahoo)

● Push model
● This method involves denormalizing the user's activity
data and pushing the meta data to all the user's friends
at the time it occurs. (from Quora)
● Pull model
● This method involves keeping all recent activity data in
memory and pulling in (or fanning out) that data at the
time a user loads their home page. Data doesn't need to
be pushed out to all subscribers as soon as it happens,
so no back-log and no disk seeks (from Quora)
● Mix model
● Active user using push model
● Non active user using pull

ZingMe News Feed system history


● First version
● Using PHP for worker
● Using MySQL for feed item
● Using MySQL for feed indexing
● Having full feature: feed type filtering, ignoring
users ..
● Restarting DB and other services are the favorite
jobs at that time :)
● Lesson learn:
– Relation DB may not be fit for this kind of project


● Second version
● Still using PHP for worker
● Using Cassandra for feed item
● Using home build list id service for feed indexing
● Using Memcached for caching item
● Removing all deluxe features :) (stupid features due to
our limited technique)
● Restarting Cassandra, and waiting for compaction is our
favorite jobs :) :)
● Headache with changing avatar
● Lesson learn: believe only ourself


● Third version
● Moving to Java for better performance
● Still using Cassandra for feed item
● Trying to use redis in Lab
● Keep only simple features (KISS)
● Cannot control memcache
– The new one expired before the old one ???
– Memcached is wrong ???
● Cannot believe to Cassandra
● Lesson learn: memcached is not the “thuốc tiên” :)

ZingMe News Feed system
(The current one :))


● Still using push model because of Twitter public some
info related to this model
● Not enough technical when choosing pull model
● Begin to understand a little bit about how to keep it
scaling
● Do not use Cassandra any more for such kind of this
system → do not believe to anyone, learn from what
they do and try our best


● Feed Item
● UserId, ObjectId, Created date...
● Storage: home build based on Kyoto Cabinet
● Fast recovery when crash
● Feed Index
● UserId → [feedId1,feedId2...]
● Storage: home build
● Fast recovery when crash


● Rate limit
● Prefilter Spam or auto tool based on rate of write request
● When hit limit, block that user for amount of time
● Feed writer
● Receive the write command
● Get the next Id from Generator
● Push the item to queue
● Return the feedId for future reference


● Gearman feed storage queue
● Very fast
● Support multi language client
● Some time block the all workers when network
unstable :)
● Solve most of our heavy jobs


● Feed Sync center
● Sync the new feed to the others such as:
– Spam detection
– Feed ranking system
– Logging system
● Feed replication function for future use


● Feed Render worker
● The main and heavy job:
– Get the feed item
– Extract the template id
– Get user info
– Render the feed based on them
● Put rendered feed in to appropriate cache
● Mobile and Desktop are totally different


● Feed Aggregate
● Get the feed index
● Get the rendered item from cache
● Return to the front-end
● Some cheat:
– If the cached items less than 5, in stead of returning
the data return a JavaScript to reload that list
– At the same time push a task to warm-up the
rendered cache
● Auto fail-over when a cache service die

Some statistics

● ~15M actions / day
● 10% Spam
● Gift receive
● Meaningless status
● Cache hit 98%
● ~80M registered users
● ~3M active users / days
● Max 1000 friends only
● Unlimited followers

Bonus
●
Twemcache (https://github.com/twitter/twemcache)
● From Twitter
● Solve most problems with memcached
● More strategy for eviction items
– Item LRU eviction: per-slabclass LRU eviction
– Random eviction : evict all items from a randomly chosen slab
– ...
● Twemcache proxy
●
Redis (http://redis.io)
● Replacement for home build when you have not enough time
● Set is default supported
● Supported cluster
● Persistence

Q&A
● What is the problem with followers?
● Do that with the trick
● Cheating the owner ;)

Q&A

Contact info:
Chau Nguyen Nhat Thanh

thanhcnn@vng.com.vn

me.zing.vn/thanhcnn2000

Building ZingMe News Feed System

More Related Content

Viewers also liked

Similar to Building ZingMe News Feed System

More from Chau Thanh

Recently uploaded

Building ZingMe News Feed System