SlideShare a Scribd company logo
Activity Feeds Architecture
                                   January, 2011




Friday, January 14, 2011
Activity Feeds Architecture




                                       To Be Covered:
                 •Data model
                 •Where feeds come from
                 •How feeds are displayed
                 •Optimizations




Friday, January 14, 2011
Activity Feeds Architecture


                                  Fundamental Entities
                            Connections                        Activities




Friday, January 14, 2011

There are two fundamental building blocks for feeds: connections and activities.
Activities form a log of what some entity on the site has done, or had done to it.
Connections express relationships between entities.
I will explain the data model for connections first.
Activity Feeds Architecture


                                           Connections


                                               Favorites
                                                                         Circles




                                                           Connections


                                        Etc.



                                                              Orders




Friday, January 14, 2011

Connections are a superset of Circles, Favorites, Orders, and other relationships between
entities on the site.
Activity Feeds Architecture

                                             Connections
                                     A                 B




                                                               C




                                         F




                                                   E           D




                                               G                   I




                                         H

                                                           J




Friday, January 14, 2011

Connections are implemented as a directed graph.
Currently, the nodes can be people or shops. (In principle they can be other objects.)
Activity Feeds Architecture


                                                      Connections
                                              A                 B




                                                                        C




                                                  F
                   connection_edges_forward

                   connection_edges_reverse                 E           D




                                                        G                   I




                                                  H

                                                                    J




Friday, January 14, 2011

The edges of the graph are stored in two tables.
For any node, connection_edges_forward lists outgoing edges and connection_edges_reverse
lists the incoming edges.
In other words, we store each edge twice.
Activity Feeds Architecture


                                               Connections
                                                             aBA
                                                                                                          aCB


                                     A                                     B

                                                                                                    aBC

                                         aFA               aEA                                                   C
                                                                           aEB
                                                                                              aDB
                                                                                                                           aCD

                                                       aEF
                                         F


                                                 aFE
                                                                       E                                         D


                                                                                        aEI
                                                                                                                     aDI




                                                                                  aIG
                                               aHE



                                                                 G                       aJG                         I

                                                     aHG                    aGJ
                                                                                                           aIJ




                                         H                           aHJ

                                                                                                J




Friday, January 14, 2011

We also assign each edge a weight, known as affinity.
Activity Feeds Architecture


                                           Connections
         On H’s shard

               connection_edges_forward


                               from                  to         affinity
                                  H                  E            0.3
                                  H                  G            0.7
                connection_edges_reverse


                               from                  to         affinity
                                   J                 H            0.75




Friday, January 14, 2011

Here we see the data for Anda’s connections on her shard.
She has two entries in the forward connections table for the people in her circle.
She has one entry in the reverse connections so that she can see everyone following her.
Activity Feeds Architecture




                                            Activities




Friday, January 14, 2011

Activities are the other database entity important to activity feeds.
Activity Feeds Architecture




                           activity := (subject, verb, object)




Friday, January 14, 2011

As you can see in Rob’s magnetic poetry diagram, activities are a description of an event on
Etsy boiled down to a subject (“who did it”), a verb (“what they did”), and an object (“what
they did it to”).
Activity Feeds Architecture




                           activity := (subject, verb, object)


                               (Steve, connected, Kyle)

                             (Kyle, favorited, brief jerky)




Friday, January 14, 2011

Here are some examples of activities.
The first one describes Steve adding Kyle to his circle.
The second one describes Kyle favoriting an item.
In each of these cases note that there are probably several parties interested in these events
[examples]. The problem (the main one we’re trying to solve with activity feeds) is how to
notify all of them about it. In order to achieve that goal, as usual we copy the data all over the
place.
Activity Feeds Architecture




                           activity := (subject, verb, object)



        activity := [owner,(subject, verb, object)]




Friday, January 14, 2011

So what we do is duplicate the S,V,O combinations with different owners.
Steve will have his record that he connected to Kyle, and Kyle will be given his own record
that Steve connected to him.
Activity Feeds Architecture


        activity := [owner,(subject, verb, object)]



                                                      [Steve, (Steve, connected, Kyle)]   Steve's shard




                           (Steve, connected, Kyle)




                                                      [Kyle, (Steve, connected, Kyle)]    Kyle's shard




Friday, January 14, 2011

This is what that looks like.
Activity Feeds Architecture


        activity := [owner,(subject, verb, object)]


                                                                [Kyle, (Kyle, favorited, brief jerky)]       Kyle's shard




                            (Kyle, favorited, brief jerky)




                                                         [MixedSpecies, (Kyle, favorited, brief jerky)]
                                                                                                          Mixedspecies' shard
                                                         [brief jerky, (Kyle, favorited, brief jerky)]




Friday, January 14, 2011

In more complicated examples there could be more than two owners.
You could envision people being interested in Kyle, people being interested in MixedSpecies,
or people being interested in brief jerky.
In cases where there are this many writes, we will generally perform them with Gearman.
Again, in order for interested parties to find the activities, we copy the activities all over the
place.
Activity Feeds Architecture




                                                   Building a Feed

                                (Kyle, favorited, brief jerky)
                                                                                       ‫ּט_ּט‬
                                 (Steve, connected, Kyle)



                                                                                         Magic,
                                                                                        cheating




                                                                  Magic,
                                                                            Newsfeed
                                                                 cheating




Friday, January 14, 2011

Now that we know about connections and activities, we can talk about how activities are
turned into Newsfeeds and how those wind up being displayed to end users.
Activity Feeds Architecture




                                                    Building a Feed

                                (Kyle, favorited, brief jerky)
                                                                                       ‫ּט_ּט‬
                                 (Steve, connected, Kyle)
                                                                                               Display

                                                                                         Magic,
                                                                                        cheating




                                                                  Magic,
                                                                            Newsfeed
                                                                 cheating



                                                     Aggregation




Friday, January 14, 2011

Getting to the end result (the activity feed page) has two distinct phases: aggregation and
display.
Activity Feeds Architecture



                                                          Aggregation

                                          (Steve, connected, Kyle)
                               Shard 1      (Steve, favorited, foo)




                                         (theblackapple, listed, widget)
                                             (Wil, bought, mittens)
                               Shard 2

                                                                           Newsfeed



                                         (Wil, bought, mittens)
                               Shard 3




                                          (Steve, connected, Kyle)
                               Shard 4   (Kyle, favorited, brief jerky)




Friday, January 14, 2011

I am going to talk about aggregation first.
Aggregation turns activities (in the database) into a Newsfeed (in memcache).
Aggregation typically occurs offline, with Gearman.
Activity Feeds Architecture


                           Aggregation, Step 1: Choosing Connections




                                                         Potentially way too many




Friday, January 14, 2011

We already allow people to have more connections than would make sense on a single feed,
or could be practically aggregated all at once.
The first step in aggregation is to turn the list of people you are connected to into the list of
people we’re actually going to go seek out activities for.
Activity Feeds Architecture


                           Aggregation, Step 1: Choosing Connections

                                                    “In Theory”


                      1


                 0.75
   Affinity
                   0.5


                 0.25


                      0
                                                    Connection



                                      $choose_connection = mt_rand() < $affinity;


Friday, January 14, 2011

In theory, the way we would do this is rank the connections by affinity and then treat the
affinity as the probability that we’ll pick it.
Activity Feeds Architecture


                           Aggregation, Step 1: Choosing Connections

                                                    “In Theory”


                      1


                 0.75
   Affinity
                   0.5


                 0.25


                      0
                                                    Connection



                                      $choose_connection = mt_rand() < $affinity;


Friday, January 14, 2011

So then we’d be more likely to pick the close connections, but leaving the possibility that we
will pick the distant ones.
Activity Feeds Architecture


                           Aggregation, Step 1: Choosing Connections

                                               “In Practice”


                      1


                 0.75
   Affinity
                   0.5


                 0.25


                      0
                                                Connection




Friday, January 14, 2011

In practice we don’t really handle affinity yet.
Activity Feeds Architecture


                           Aggregation, Step 1: Choosing Connections

                                            “Even More In Practice”


                      1


                 0.75
   Affinity
                   0.5


                 0.25


                      0

                                                  Connection




Friday, January 14, 2011

And most people don’t currently have enough connections for this to matter at all. (Mean
connections is around a dozen.)
Activity Feeds Architecture


                           Aggregation, Step 2: Making Activity Sets

                                          score      activity   activity   activity   activity

                                                       0.0        0.0        0.0        0.0
                                          flags
                                                      0x0        0x0        0x0        0x0




                                                     activity   activity

                                                       0.0        0.0
                                                      0x0        0x0




                                                     activity   activity   activity

                                                       0.0        0.0        0.0
                                                      0x0        0x0        0x0




                                                     activity   activity   activity   activity

                                                       0.0        0.0        0.0        0.0
                                                       0x0        0x0        0x0        0x0




Friday, January 14, 2011

Once the connections are chosen, we then select historical activity for them and convert them
into in-memory structures called activity sets.
These are just the activities grouped by connection, with a score and flags field for each.
The next few phases of aggregation operate on these.
Activity Feeds Architecture


                              Aggregation, Step 3: Classification

                                            activity   activity   activity   activity

                                              0.0        0.0        0.0        0.0
                                             0x11       0x3       0x20c1     0x10004




                                            activity   activity

                                              0.0        0.0
                                            0x20c1      0x4




                                            activity   activity   activity

                                              0.0        0.0        0.0
                                            0x1001     0x2003      0x11




                                            activity   activity   activity   activity

                                              0.0        0.0        0.0        0.0
                                             0x11      0x401        0x5      0x10004




Friday, January 14, 2011

The next thing that happens is that we iterate through all of the activities in all of the sets and
classify them.
Activity Feeds Architecture


                              Aggregation, Step 3: Classification

                               activity   activity   activity   activity

                                 0.0        0.0        0.0        0.0
                                0x11       0x3       0x20c1     0x10004    about_owner_shop | user_created_treasury

                                                                           Rob created a treasury featuring
                                                                           the feed owner's shop.

                               activity   activity

                                 0.0        0.0
                               0x20c1      0x4




                               activity   activity   activity

                                 0.0        0.0        0.0
                               0x1001     0x2003      0x11




                               activity   activity   activity   activity

                                 0.0        0.0        0.0        0.0
                                0x11      0x401        0x5      0x10004




Friday, January 14, 2011

The flags are a bit field.
They are all from the point of view of the feed owner.
So the same activities on another person’s feed would be assigned different flags.
Activity Feeds Architecture


                                  Aggregation, Step 4: Scoring

                                            activity   activity   activity   activity

                                              0.8       0.77        0.9        0.1
                                             0x11       0x3       0x20c1     0x10004




                                            activity   activity

                                              0.6       0.47
                                            0x20c1      0x4




                                            activity   activity   activity

                                              0.8       0.55        0.8
                                            0x1001     0x2003      0x11




                                            activity   activity   activity   activity

                                              0.3       0.25       0.74        0.9
                                             0x11      0x401        0x5      0x10004




Friday, January 14, 2011

Next we fill in the score fields.
At this point the score is just a simple time decay function (older activities always score lower
than new ones).
Activity Feeds Architecture


                                  Aggregation, Step 5: Pruning
                                                                               [Rob, (Rob, connected, Jared)]



                                   activity   activity   activity   activity

                                     0.8       0.77        0.9        0.1
                                    0x11       0x3       0x20c1     0x10004




                                   activity   activity

                                     0.6       0.47
                                   0x20c1      0x4




                                   activity   activity   activity              [Jared, (Rob, connected, Jared)]

                                     0.8       0.55        0.8
                                   0x1001     0x2003      0x11




                                   activity   activity   activity   activity

                                     0.3       0.25       0.74        0.9
                                    0x11      0x401        0x5      0x10004




Friday, January 14, 2011

As we noted before it’s possible to wind up seeing the same event as two or more activities.
The next stage of aggregation detects these situations.
Activity Feeds Architecture


                                  Aggregation, Step 5: Pruning
                                                                               [Rob, (Rob, connected, Jared)]



                                   activity   activity   activity   activity

                                     0.8       0.77        0.9        0.1
                                    0x11       0x3       0x20c1     0x10004




                                   activity   activity

                                     0.6       0.47
                                   0x20c1      0x4




                                   activity   activity   activity              [Jared, (Rob, connected, Jared)]

                                     0.8       0.55        0.8
                                   0x1001     0x2003      0x11




                                   activity   activity   activity   activity

                                     0.3       0.25       0.74        0.9
                                    0x11      0x401        0x5      0x10004




Friday, January 14, 2011

We iterate through the activity sets and remove the duplicates.
Right now we can just cross off the second instance of the SVO pair; once we have comments
this will be more complicated.
Activity Feeds Architecture


                                                  Aggregation, Step 6: Sort & Merge

                           activity   activity   activity   activity

                             0.8       0.77        0.9        0.1
                            0x11       0x3       0x20c1     0x10004




                           activity   activity

                             0.6       0.47
                           0x20c1      0x4
                                                                       activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity

                                                                         0.9        0.9        0.8        0.8       0.77       0.74        0.6       0.55       0.47        0.3       0.25        0.1
                                                                       0x20c1     0x10004    0x1001      0x11       0x3        0x5       0x20c1     0x2003      0x4        0x11      0x401      0x10004
                           activity   activity

                             0.8       0.55
                           0x1001     0x2003




                           activity   activity   activity   activity

                             0.3       0.25       0.74        0.9
                            0x11      0x401        0x5      0x10004




Friday, January 14, 2011

Once everything is scored, classified, and de-duped we can flatten the whole thing and sort it
by score.
Activity Feeds Architecture


                                                       Aggregation, Step 6: Sort & Merge

                                           activity    activity    activity   activity   activity   activity   activity   activity   activity   activity   activity   activity

                                             0.9         0.9         0.8        0.8       0.77       0.74        0.6       0.55       0.47        0.3       0.25         0.1
                                           0x20c1      0x10004     0x1001      0x11       0x3        0x5       0x20c1     0x2003      0x4        0x11      0x401      0x10004




         activity   activity    activity    activity    activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity

           0.9        0.9         0.8         0.8        0.77       0.74      0.716       0.71        0.6        0.6       0.55       0.47        0.3        0.3      0.291       0.25        0.1      0.097       0.02
         0x20c1     0x10004     0x1001       0x11        0x3        0x5       0x4c01     0x2c01     0x20c1     0xc001     0x2003      0x4        0x11      0x4001     0x4001     0x401      0x10004     0x4       0x1001



        Newsfeed




Friday, January 14, 2011

Then we take the final set of activities and merge it on to the owner’s existing newsfeed.
(Or we create a new newsfeed if they don’t have one.)
Activity Feeds Architecture


                                                                  Aggregation: Cleaning Up



                                                                                                                                                   Just fine            Too many


          activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity

            0.9        0.9        0.8        0.8       0.77       0.74      0.716       0.71        0.6        0.6       0.55       0.47        0.3        0.3      0.291       0.25        0.1      0.097       0.02
          0x20c1     0x10004    0x1001      0x11       0x3        0x5       0x4c01     0x2c01     0x20c1     0xc001     0x2003      0x4        0x11      0x4001     0x4001     0x401      0x10004     0x4       0x1001



         Newsfeed




Friday, January 14, 2011

We trim off the end of the newsfeed, so that they don’t become arbitrarily large.
Activity Feeds Architecture


                                                            Aggregation: Cleaning Up



                           activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity   activity

                             0.9        0.9        0.8        0.8       0.77       0.74      0.716       0.71        0.6        0.6       0.55       0.47        0.3        0.3
                           0x20c1     0x10004    0x1001      0x11       0x3        0x5       0x4c01     0x2c01     0x20c1     0xc001     0x2003      0x4        0x11      0x4001



                           Newsfeed




                                                                                             memcached




Friday, January 14, 2011

And then finally we stuff the feed into memcached.
Activity Feeds Architecture


                                            Aggregation




Friday, January 14, 2011

Currently we peak at doing this about 25 times per second.
Activity Feeds Architecture


                                            Aggregation




Friday, January 14, 2011

We do it for a lot of different reasons:
- The feed owner has done something, or logged in.
- On a schedule, with cron.
- We also aggregate for your connections when you do something (purple). This is a hack and
won’t scale.
Activity Feeds Architecture



                                              Display




                 memcached




Friday, January 14, 2011

Next I’m going to talk about how we get from the memcached newsfeed to the final product.
Activity Feeds Architecture



                            Display: done naively, sucks




Friday, January 14, 2011

If we just showed the activities in the order that they’re in on the newsfeed, it would look like
this.
Activity Feeds Architecture



                                 Display: Enter Rollups




Friday, January 14, 2011

To solve this problem we combine similar stories into rollups.
Activity Feeds Architecture



                             Display: Computing Rollups




Friday, January 14, 2011

Here’s an attempt at depicting how rollups are created.
The feed is divided up into sections, so that you don’t wind up seeing all of the reds, greens,
etc. on the entire feed in just a few very large rollups.
Then the similar stories are grouped together within the sections.
Activity Feeds Architecture



                                           Display: Filling in Stories


                           activity                                                            Story
                                                           Story
                             0.9
                                                                                                                      html
                                                                                           (Feed-owner-
                                      StoryHydrator   (Global details)      StoryTeller                      Smarty
                           0x10004                                                        specific details)




                                                                         memcached




Friday, January 14, 2011

Once that’s done, we can go through the rest of the display pipeline for the root story in each
rollup.
There are multiple layers of caching here. Things that are global (like the shop associated
with a favorited listing) are cached separately from things that are unique to the person
looking at the feed (like the exact way the story is phrased).
Activity Feeds Architecture



                                         Making it Fast
                                                Response Time (ms)

             1200


                900

                                                                              Boom
                600


                300


                    0
                           Homepage      Shop          Listing       Search     Activity



Friday, January 14, 2011

Finally I’m going to go through a few ways that we’ve sped up activity, to the point where it’s
one of the faster pages on the site (despite being pretty complicated).
Activity Feeds Architecture



                               Hack #1: Cache Warming
                                                                                       !_!

                                (Kyle, favorited, brief jerky)
                                 (Steve, connected, Kyle)


                                                                                        Magic,
                                                                                       cheating




                                                                  Magic,
                                                                            Newsfeed
                                                                 cheating




Friday, January 14, 2011

The first thing we do to speed things up is run almost the entire pipeline proactively using
gearman.
So after aggregation we trigger a display run, even though nobody is there to look at the
html.
The end result is that almost every pageview is against a hot cache.
Activity Feeds Architecture



                                 Hack #2: TTL Caching

     May be his avatar
      from 5 minutes
           ago.

         Big f’ing deal.




Friday, January 14, 2011

The second thing we do is add bits of TTL caching where few people will notice.
Straightforward but not done in many places on the site.
Note that his avatar here is tied to the story. If he generates new activity he’ll see his new
avatar.
Activity Feeds Architecture



                           Hack #3: Judicious Associations

      getFinder(“UserProfile”)->find
                   (...)

                       not
       getFinder(“User”)->find(...)-
                 >Profile




Friday, January 14, 2011

We also profiled the pages and meticulously simplified ORM usage.
Again this sounds obvious but it’s really easy to lose track of what you’re doing as you hand
the user off to the template. Lots of ORM calls were originally actually being made by the
template.
Activity Feeds Architecture



                           Hack #4: Lazy Below the Fold




                            We don’t load much at the outset.
                            You get more as you scroll down
                            (finite scrolling).


Friday, January 14, 2011
The End




Friday, January 14, 2011

More Related Content

What's hot

Introduction to Storm
Introduction to Storm Introduction to Storm
Introduction to Storm
Chandler Huang
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
Saurav Haloi
 
From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.
Taras Matyashovsky
 
Solving PostgreSQL wicked problems
Solving PostgreSQL wicked problemsSolving PostgreSQL wicked problems
Solving PostgreSQL wicked problems
Alexander Korotkov
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
Jun Rao
 
Scalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability Patterns
Jonas Bonér
 
Spark shuffle introduction
Spark shuffle introductionSpark shuffle introduction
Spark shuffle introduction
colorant
 
MongoDB Fundamentals
MongoDB FundamentalsMongoDB Fundamentals
MongoDB Fundamentals
MongoDB
 
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraApache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native Era
Flink Forward
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
Databricks
 
Deploying Flink on Kubernetes - David Anderson
 Deploying Flink on Kubernetes - David Anderson Deploying Flink on Kubernetes - David Anderson
Deploying Flink on Kubernetes - David Anderson
Ververica
 
Introduction to DataFusion An Embeddable Query Engine Written in Rust
Introduction to DataFusion  An Embeddable Query Engine Written in RustIntroduction to DataFusion  An Embeddable Query Engine Written in Rust
Introduction to DataFusion An Embeddable Query Engine Written in Rust
Andrew Lamb
 
Hive, Presto, and Spark on TPC-DS benchmark
Hive, Presto, and Spark on TPC-DS benchmarkHive, Presto, and Spark on TPC-DS benchmark
Hive, Presto, and Spark on TPC-DS benchmark
Dongwon Kim
 
Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison
Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison
Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison
Severalnines
 
Apache ZooKeeper
Apache ZooKeeperApache ZooKeeper
Apache ZooKeeper
Scott Leberknight
 
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and HudiHow to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
Flink Forward
 
Virtual Nodes: Rethinking Topology in Cassandra
Virtual Nodes: Rethinking Topology in CassandraVirtual Nodes: Rethinking Topology in Cassandra
Virtual Nodes: Rethinking Topology in Cassandra
Eric Evans
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
Arnab Mitra
 
Apache Arrow Flight Overview
Apache Arrow Flight OverviewApache Arrow Flight Overview
Apache Arrow Flight Overview
Jacques Nadeau
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
Flink Forward
 

What's hot (20)

Introduction to Storm
Introduction to Storm Introduction to Storm
Introduction to Storm
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
 
From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.
 
Solving PostgreSQL wicked problems
Solving PostgreSQL wicked problemsSolving PostgreSQL wicked problems
Solving PostgreSQL wicked problems
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
 
Scalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability Patterns
 
Spark shuffle introduction
Spark shuffle introductionSpark shuffle introduction
Spark shuffle introduction
 
MongoDB Fundamentals
MongoDB FundamentalsMongoDB Fundamentals
MongoDB Fundamentals
 
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraApache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native Era
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
 
Deploying Flink on Kubernetes - David Anderson
 Deploying Flink on Kubernetes - David Anderson Deploying Flink on Kubernetes - David Anderson
Deploying Flink on Kubernetes - David Anderson
 
Introduction to DataFusion An Embeddable Query Engine Written in Rust
Introduction to DataFusion  An Embeddable Query Engine Written in RustIntroduction to DataFusion  An Embeddable Query Engine Written in Rust
Introduction to DataFusion An Embeddable Query Engine Written in Rust
 
Hive, Presto, and Spark on TPC-DS benchmark
Hive, Presto, and Spark on TPC-DS benchmarkHive, Presto, and Spark on TPC-DS benchmark
Hive, Presto, and Spark on TPC-DS benchmark
 
Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison
Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison
Galera Cluster for MySQL vs MySQL (NDB) Cluster: A High Level Comparison
 
Apache ZooKeeper
Apache ZooKeeperApache ZooKeeper
Apache ZooKeeper
 
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and HudiHow to build a streaming Lakehouse with Flink, Kafka, and Hudi
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
 
Virtual Nodes: Rethinking Topology in Cassandra
Virtual Nodes: Rethinking Topology in CassandraVirtual Nodes: Rethinking Topology in Cassandra
Virtual Nodes: Rethinking Topology in Cassandra
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Apache Arrow Flight Overview
Apache Arrow Flight OverviewApache Arrow Flight Overview
Apache Arrow Flight Overview
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
 

Viewers also liked

Marketing 2.0 - L'utente come canale di vendita grazie all'approccio conversa...
Marketing 2.0 - L'utente come canale di vendita grazie all'approccio conversa...Marketing 2.0 - L'utente come canale di vendita grazie all'approccio conversa...
Marketing 2.0 - L'utente come canale di vendita grazie all'approccio conversa...
Claudio Vaccaro
 
Copy Strategy Brand Amiacque
Copy Strategy Brand AmiacqueCopy Strategy Brand Amiacque
Copy Strategy Brand Amiacque
Nopeidea ®
 
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Zalo_app
 
Sơ lược kiến trúc hệ thống Zing Me
Sơ lược kiến trúc hệ thống Zing MeSơ lược kiến trúc hệ thống Zing Me
Sơ lược kiến trúc hệ thống Zing Me
zingopen
 
Building ZingMe News Feed System
Building ZingMe News Feed SystemBuilding ZingMe News Feed System
Building ZingMe News Feed System
Chau Thanh
 
Design a scalable social network: Problems and solutions
Design a scalable social network: Problems and solutionsDesign a scalable social network: Problems and solutions
Design a scalable social network: Problems and solutions
Chau Thanh
 
Choose Boring Technology
Choose Boring TechnologyChoose Boring Technology
Choose Boring Technology
Dan McKinley
 
facebook architecture for 600M users
facebook architecture for 600M usersfacebook architecture for 600M users
facebook architecture for 600M users
Jongyoon Choi
 
Design for Continuous Experimentation
Design for Continuous ExperimentationDesign for Continuous Experimentation
Design for Continuous Experimentation
Dan McKinley
 
The Brand Strategy Canvas: a One-Page Strategy for Startups
The Brand Strategy Canvas: a One-Page Strategy for StartupsThe Brand Strategy Canvas: a One-Page Strategy for Startups
The Brand Strategy Canvas: a One-Page Strategy for Startups
patrickjwoods
 

Viewers also liked (11)

14 La Copy Strategy
14 La Copy Strategy14 La Copy Strategy
14 La Copy Strategy
 
Marketing 2.0 - L'utente come canale di vendita grazie all'approccio conversa...
Marketing 2.0 - L'utente come canale di vendita grazie all'approccio conversa...Marketing 2.0 - L'utente come canale di vendita grazie all'approccio conversa...
Marketing 2.0 - L'utente come canale di vendita grazie all'approccio conversa...
 
Copy Strategy Brand Amiacque
Copy Strategy Brand AmiacqueCopy Strategy Brand Amiacque
Copy Strategy Brand Amiacque
 
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
Inside Zalo: Developing a mobile messenger for the audience of millions - VN ...
 
Sơ lược kiến trúc hệ thống Zing Me
Sơ lược kiến trúc hệ thống Zing MeSơ lược kiến trúc hệ thống Zing Me
Sơ lược kiến trúc hệ thống Zing Me
 
Building ZingMe News Feed System
Building ZingMe News Feed SystemBuilding ZingMe News Feed System
Building ZingMe News Feed System
 
Design a scalable social network: Problems and solutions
Design a scalable social network: Problems and solutionsDesign a scalable social network: Problems and solutions
Design a scalable social network: Problems and solutions
 
Choose Boring Technology
Choose Boring TechnologyChoose Boring Technology
Choose Boring Technology
 
facebook architecture for 600M users
facebook architecture for 600M usersfacebook architecture for 600M users
facebook architecture for 600M users
 
Design for Continuous Experimentation
Design for Continuous ExperimentationDesign for Continuous Experimentation
Design for Continuous Experimentation
 
The Brand Strategy Canvas: a One-Page Strategy for Startups
The Brand Strategy Canvas: a One-Page Strategy for StartupsThe Brand Strategy Canvas: a One-Page Strategy for Startups
The Brand Strategy Canvas: a One-Page Strategy for Startups
 

Recently uploaded

Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
BibashShahi
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
Miro Wengner
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
DianaGray10
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
Javier Junquera
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
FilipTomaszewski5
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
christinelarrosa
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Fwdays
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 

Recently uploaded (20)

Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
 
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green MasterplanJavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
What is an RPA CoE? Session 2 – CoE Roles
What is an RPA CoE?  Session 2 – CoE RolesWhat is an RPA CoE?  Session 2 – CoE Roles
What is an RPA CoE? Session 2 – CoE Roles
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)GNSS spoofing via SDR (Criptored Talks 2024)
GNSS spoofing via SDR (Criptored Talks 2024)
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 

Etsy Activity Feeds Architecture

  • 1. Activity Feeds Architecture January, 2011 Friday, January 14, 2011
  • 2. Activity Feeds Architecture To Be Covered: •Data model •Where feeds come from •How feeds are displayed •Optimizations Friday, January 14, 2011
  • 3. Activity Feeds Architecture Fundamental Entities Connections Activities Friday, January 14, 2011 There are two fundamental building blocks for feeds: connections and activities. Activities form a log of what some entity on the site has done, or had done to it. Connections express relationships between entities. I will explain the data model for connections first.
  • 4. Activity Feeds Architecture Connections Favorites Circles Connections Etc. Orders Friday, January 14, 2011 Connections are a superset of Circles, Favorites, Orders, and other relationships between entities on the site.
  • 5. Activity Feeds Architecture Connections A B C F E D G I H J Friday, January 14, 2011 Connections are implemented as a directed graph. Currently, the nodes can be people or shops. (In principle they can be other objects.)
  • 6. Activity Feeds Architecture Connections A B C F connection_edges_forward connection_edges_reverse E D G I H J Friday, January 14, 2011 The edges of the graph are stored in two tables. For any node, connection_edges_forward lists outgoing edges and connection_edges_reverse lists the incoming edges. In other words, we store each edge twice.
  • 7. Activity Feeds Architecture Connections aBA aCB A B aBC aFA aEA C aEB aDB aCD aEF F aFE E D aEI aDI aIG aHE G aJG I aHG aGJ aIJ H aHJ J Friday, January 14, 2011 We also assign each edge a weight, known as affinity.
  • 8. Activity Feeds Architecture Connections On H’s shard connection_edges_forward from to affinity H E 0.3 H G 0.7 connection_edges_reverse from to affinity J H 0.75 Friday, January 14, 2011 Here we see the data for Anda’s connections on her shard. She has two entries in the forward connections table for the people in her circle. She has one entry in the reverse connections so that she can see everyone following her.
  • 9. Activity Feeds Architecture Activities Friday, January 14, 2011 Activities are the other database entity important to activity feeds.
  • 10. Activity Feeds Architecture activity := (subject, verb, object) Friday, January 14, 2011 As you can see in Rob’s magnetic poetry diagram, activities are a description of an event on Etsy boiled down to a subject (“who did it”), a verb (“what they did”), and an object (“what they did it to”).
  • 11. Activity Feeds Architecture activity := (subject, verb, object) (Steve, connected, Kyle) (Kyle, favorited, brief jerky) Friday, January 14, 2011 Here are some examples of activities. The first one describes Steve adding Kyle to his circle. The second one describes Kyle favoriting an item. In each of these cases note that there are probably several parties interested in these events [examples]. The problem (the main one we’re trying to solve with activity feeds) is how to notify all of them about it. In order to achieve that goal, as usual we copy the data all over the place.
  • 12. Activity Feeds Architecture activity := (subject, verb, object) activity := [owner,(subject, verb, object)] Friday, January 14, 2011 So what we do is duplicate the S,V,O combinations with different owners. Steve will have his record that he connected to Kyle, and Kyle will be given his own record that Steve connected to him.
  • 13. Activity Feeds Architecture activity := [owner,(subject, verb, object)] [Steve, (Steve, connected, Kyle)] Steve's shard (Steve, connected, Kyle) [Kyle, (Steve, connected, Kyle)] Kyle's shard Friday, January 14, 2011 This is what that looks like.
  • 14. Activity Feeds Architecture activity := [owner,(subject, verb, object)] [Kyle, (Kyle, favorited, brief jerky)] Kyle's shard (Kyle, favorited, brief jerky) [MixedSpecies, (Kyle, favorited, brief jerky)] Mixedspecies' shard [brief jerky, (Kyle, favorited, brief jerky)] Friday, January 14, 2011 In more complicated examples there could be more than two owners. You could envision people being interested in Kyle, people being interested in MixedSpecies, or people being interested in brief jerky. In cases where there are this many writes, we will generally perform them with Gearman. Again, in order for interested parties to find the activities, we copy the activities all over the place.
  • 15. Activity Feeds Architecture Building a Feed (Kyle, favorited, brief jerky) ‫ּט_ּט‬ (Steve, connected, Kyle) Magic, cheating Magic, Newsfeed cheating Friday, January 14, 2011 Now that we know about connections and activities, we can talk about how activities are turned into Newsfeeds and how those wind up being displayed to end users.
  • 16. Activity Feeds Architecture Building a Feed (Kyle, favorited, brief jerky) ‫ּט_ּט‬ (Steve, connected, Kyle) Display Magic, cheating Magic, Newsfeed cheating Aggregation Friday, January 14, 2011 Getting to the end result (the activity feed page) has two distinct phases: aggregation and display.
  • 17. Activity Feeds Architecture Aggregation (Steve, connected, Kyle) Shard 1 (Steve, favorited, foo) (theblackapple, listed, widget) (Wil, bought, mittens) Shard 2 Newsfeed (Wil, bought, mittens) Shard 3 (Steve, connected, Kyle) Shard 4 (Kyle, favorited, brief jerky) Friday, January 14, 2011 I am going to talk about aggregation first. Aggregation turns activities (in the database) into a Newsfeed (in memcache). Aggregation typically occurs offline, with Gearman.
  • 18. Activity Feeds Architecture Aggregation, Step 1: Choosing Connections Potentially way too many Friday, January 14, 2011 We already allow people to have more connections than would make sense on a single feed, or could be practically aggregated all at once. The first step in aggregation is to turn the list of people you are connected to into the list of people we’re actually going to go seek out activities for.
  • 19. Activity Feeds Architecture Aggregation, Step 1: Choosing Connections “In Theory” 1 0.75 Affinity 0.5 0.25 0 Connection $choose_connection = mt_rand() < $affinity; Friday, January 14, 2011 In theory, the way we would do this is rank the connections by affinity and then treat the affinity as the probability that we’ll pick it.
  • 20. Activity Feeds Architecture Aggregation, Step 1: Choosing Connections “In Theory” 1 0.75 Affinity 0.5 0.25 0 Connection $choose_connection = mt_rand() < $affinity; Friday, January 14, 2011 So then we’d be more likely to pick the close connections, but leaving the possibility that we will pick the distant ones.
  • 21. Activity Feeds Architecture Aggregation, Step 1: Choosing Connections “In Practice” 1 0.75 Affinity 0.5 0.25 0 Connection Friday, January 14, 2011 In practice we don’t really handle affinity yet.
  • 22. Activity Feeds Architecture Aggregation, Step 1: Choosing Connections “Even More In Practice” 1 0.75 Affinity 0.5 0.25 0 Connection Friday, January 14, 2011 And most people don’t currently have enough connections for this to matter at all. (Mean connections is around a dozen.)
  • 23. Activity Feeds Architecture Aggregation, Step 2: Making Activity Sets score activity activity activity activity 0.0 0.0 0.0 0.0 flags 0x0 0x0 0x0 0x0 activity activity 0.0 0.0 0x0 0x0 activity activity activity 0.0 0.0 0.0 0x0 0x0 0x0 activity activity activity activity 0.0 0.0 0.0 0.0 0x0 0x0 0x0 0x0 Friday, January 14, 2011 Once the connections are chosen, we then select historical activity for them and convert them into in-memory structures called activity sets. These are just the activities grouped by connection, with a score and flags field for each. The next few phases of aggregation operate on these.
  • 24. Activity Feeds Architecture Aggregation, Step 3: Classification activity activity activity activity 0.0 0.0 0.0 0.0 0x11 0x3 0x20c1 0x10004 activity activity 0.0 0.0 0x20c1 0x4 activity activity activity 0.0 0.0 0.0 0x1001 0x2003 0x11 activity activity activity activity 0.0 0.0 0.0 0.0 0x11 0x401 0x5 0x10004 Friday, January 14, 2011 The next thing that happens is that we iterate through all of the activities in all of the sets and classify them.
  • 25. Activity Feeds Architecture Aggregation, Step 3: Classification activity activity activity activity 0.0 0.0 0.0 0.0 0x11 0x3 0x20c1 0x10004 about_owner_shop | user_created_treasury Rob created a treasury featuring the feed owner's shop. activity activity 0.0 0.0 0x20c1 0x4 activity activity activity 0.0 0.0 0.0 0x1001 0x2003 0x11 activity activity activity activity 0.0 0.0 0.0 0.0 0x11 0x401 0x5 0x10004 Friday, January 14, 2011 The flags are a bit field. They are all from the point of view of the feed owner. So the same activities on another person’s feed would be assigned different flags.
  • 26. Activity Feeds Architecture Aggregation, Step 4: Scoring activity activity activity activity 0.8 0.77 0.9 0.1 0x11 0x3 0x20c1 0x10004 activity activity 0.6 0.47 0x20c1 0x4 activity activity activity 0.8 0.55 0.8 0x1001 0x2003 0x11 activity activity activity activity 0.3 0.25 0.74 0.9 0x11 0x401 0x5 0x10004 Friday, January 14, 2011 Next we fill in the score fields. At this point the score is just a simple time decay function (older activities always score lower than new ones).
  • 27. Activity Feeds Architecture Aggregation, Step 5: Pruning [Rob, (Rob, connected, Jared)] activity activity activity activity 0.8 0.77 0.9 0.1 0x11 0x3 0x20c1 0x10004 activity activity 0.6 0.47 0x20c1 0x4 activity activity activity [Jared, (Rob, connected, Jared)] 0.8 0.55 0.8 0x1001 0x2003 0x11 activity activity activity activity 0.3 0.25 0.74 0.9 0x11 0x401 0x5 0x10004 Friday, January 14, 2011 As we noted before it’s possible to wind up seeing the same event as two or more activities. The next stage of aggregation detects these situations.
  • 28. Activity Feeds Architecture Aggregation, Step 5: Pruning [Rob, (Rob, connected, Jared)] activity activity activity activity 0.8 0.77 0.9 0.1 0x11 0x3 0x20c1 0x10004 activity activity 0.6 0.47 0x20c1 0x4 activity activity activity [Jared, (Rob, connected, Jared)] 0.8 0.55 0.8 0x1001 0x2003 0x11 activity activity activity activity 0.3 0.25 0.74 0.9 0x11 0x401 0x5 0x10004 Friday, January 14, 2011 We iterate through the activity sets and remove the duplicates. Right now we can just cross off the second instance of the SVO pair; once we have comments this will be more complicated.
  • 29. Activity Feeds Architecture Aggregation, Step 6: Sort & Merge activity activity activity activity 0.8 0.77 0.9 0.1 0x11 0x3 0x20c1 0x10004 activity activity 0.6 0.47 0x20c1 0x4 activity activity activity activity activity activity activity activity activity activity activity activity 0.9 0.9 0.8 0.8 0.77 0.74 0.6 0.55 0.47 0.3 0.25 0.1 0x20c1 0x10004 0x1001 0x11 0x3 0x5 0x20c1 0x2003 0x4 0x11 0x401 0x10004 activity activity 0.8 0.55 0x1001 0x2003 activity activity activity activity 0.3 0.25 0.74 0.9 0x11 0x401 0x5 0x10004 Friday, January 14, 2011 Once everything is scored, classified, and de-duped we can flatten the whole thing and sort it by score.
  • 30. Activity Feeds Architecture Aggregation, Step 6: Sort & Merge activity activity activity activity activity activity activity activity activity activity activity activity 0.9 0.9 0.8 0.8 0.77 0.74 0.6 0.55 0.47 0.3 0.25 0.1 0x20c1 0x10004 0x1001 0x11 0x3 0x5 0x20c1 0x2003 0x4 0x11 0x401 0x10004 activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity 0.9 0.9 0.8 0.8 0.77 0.74 0.716 0.71 0.6 0.6 0.55 0.47 0.3 0.3 0.291 0.25 0.1 0.097 0.02 0x20c1 0x10004 0x1001 0x11 0x3 0x5 0x4c01 0x2c01 0x20c1 0xc001 0x2003 0x4 0x11 0x4001 0x4001 0x401 0x10004 0x4 0x1001 Newsfeed Friday, January 14, 2011 Then we take the final set of activities and merge it on to the owner’s existing newsfeed. (Or we create a new newsfeed if they don’t have one.)
  • 31. Activity Feeds Architecture Aggregation: Cleaning Up Just fine Too many activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity activity 0.9 0.9 0.8 0.8 0.77 0.74 0.716 0.71 0.6 0.6 0.55 0.47 0.3 0.3 0.291 0.25 0.1 0.097 0.02 0x20c1 0x10004 0x1001 0x11 0x3 0x5 0x4c01 0x2c01 0x20c1 0xc001 0x2003 0x4 0x11 0x4001 0x4001 0x401 0x10004 0x4 0x1001 Newsfeed Friday, January 14, 2011 We trim off the end of the newsfeed, so that they don’t become arbitrarily large.
  • 32. Activity Feeds Architecture Aggregation: Cleaning Up activity activity activity activity activity activity activity activity activity activity activity activity activity activity 0.9 0.9 0.8 0.8 0.77 0.74 0.716 0.71 0.6 0.6 0.55 0.47 0.3 0.3 0x20c1 0x10004 0x1001 0x11 0x3 0x5 0x4c01 0x2c01 0x20c1 0xc001 0x2003 0x4 0x11 0x4001 Newsfeed memcached Friday, January 14, 2011 And then finally we stuff the feed into memcached.
  • 33. Activity Feeds Architecture Aggregation Friday, January 14, 2011 Currently we peak at doing this about 25 times per second.
  • 34. Activity Feeds Architecture Aggregation Friday, January 14, 2011 We do it for a lot of different reasons: - The feed owner has done something, or logged in. - On a schedule, with cron. - We also aggregate for your connections when you do something (purple). This is a hack and won’t scale.
  • 35. Activity Feeds Architecture Display memcached Friday, January 14, 2011 Next I’m going to talk about how we get from the memcached newsfeed to the final product.
  • 36. Activity Feeds Architecture Display: done naively, sucks Friday, January 14, 2011 If we just showed the activities in the order that they’re in on the newsfeed, it would look like this.
  • 37. Activity Feeds Architecture Display: Enter Rollups Friday, January 14, 2011 To solve this problem we combine similar stories into rollups.
  • 38. Activity Feeds Architecture Display: Computing Rollups Friday, January 14, 2011 Here’s an attempt at depicting how rollups are created. The feed is divided up into sections, so that you don’t wind up seeing all of the reds, greens, etc. on the entire feed in just a few very large rollups. Then the similar stories are grouped together within the sections.
  • 39. Activity Feeds Architecture Display: Filling in Stories activity Story Story 0.9 html (Feed-owner- StoryHydrator (Global details) StoryTeller Smarty 0x10004 specific details) memcached Friday, January 14, 2011 Once that’s done, we can go through the rest of the display pipeline for the root story in each rollup. There are multiple layers of caching here. Things that are global (like the shop associated with a favorited listing) are cached separately from things that are unique to the person looking at the feed (like the exact way the story is phrased).
  • 40. Activity Feeds Architecture Making it Fast Response Time (ms) 1200 900 Boom 600 300 0 Homepage Shop Listing Search Activity Friday, January 14, 2011 Finally I’m going to go through a few ways that we’ve sped up activity, to the point where it’s one of the faster pages on the site (despite being pretty complicated).
  • 41. Activity Feeds Architecture Hack #1: Cache Warming !_! (Kyle, favorited, brief jerky) (Steve, connected, Kyle) Magic, cheating Magic, Newsfeed cheating Friday, January 14, 2011 The first thing we do to speed things up is run almost the entire pipeline proactively using gearman. So after aggregation we trigger a display run, even though nobody is there to look at the html. The end result is that almost every pageview is against a hot cache.
  • 42. Activity Feeds Architecture Hack #2: TTL Caching May be his avatar from 5 minutes ago. Big f’ing deal. Friday, January 14, 2011 The second thing we do is add bits of TTL caching where few people will notice. Straightforward but not done in many places on the site. Note that his avatar here is tied to the story. If he generates new activity he’ll see his new avatar.
  • 43. Activity Feeds Architecture Hack #3: Judicious Associations getFinder(“UserProfile”)->find (...) not getFinder(“User”)->find(...)- >Profile Friday, January 14, 2011 We also profiled the pages and meticulously simplified ORM usage. Again this sounds obvious but it’s really easy to lose track of what you’re doing as you hand the user off to the template. Lots of ORM calls were originally actually being made by the template.
  • 44. Activity Feeds Architecture Hack #4: Lazy Below the Fold We don’t load much at the outset. You get more as you scroll down (finite scrolling). Friday, January 14, 2011