Scalable News Feed with Mongo DB

Thierry Schellenbach
Thierry SchellenbachCo-Founder & CEO at getstream.io
MongoDB Activity Feeds
Activity Feeds
Feeds: 



Follow anything (users, sites,
stocks, bands, companies,
shops, etc.,)
What is a feed?
MongoDB Activity Feed
Activity Feeds
Created this while working on a blogpost
https://github.com/GetStream/mongodb-activity-feed
The API
Activity Feeds
Initializing the feed manager
The API - Timeline Feed
Activity Feeds
The API - Notification Feed
Activity Feeds
The API - Ranking Methods
Activity Feeds
Architecture
Activity Feeds
• Unordered Bulk Writes
• Conflict-free replicated Data
Types
• Bull + Redis for Queuing
CRDTs
Activity Feeds
You might be wondering why we’re using an operation log instead of just storing the
activities. If you simply add and remove activities you run into problems with locking. IE.
a user likes something, unlikes it and likes it again. If you’re just adding and removing
activities you end up with either:
• Incorrect state (Something shows as liked when you unliked it)

• Locks (i.e. which reduce the write capacity of your system)

The best way to solve this is using CRDTs. Instead of adding or removing the activity you
store that the activity should be added/removed together with the time of the operation.
This approach prevents race conditions and doesn't require locking.
Schema
Activity Feeds
• Activity: The actual data of the activity

• Activity Feed: An operation log. This stores that
an activity should be added to a certain feed.

• Feed Group: A group of feeds, for example: user,
notification, timeline etc.

• Feed: A specific feed, ie user:scott,
notification:josh, timeline:tom, etc.
• Follow: A follow relationships between 2 feeds.
MongoDB Activity Feed
Activity Feeds
Repo:
https://github.com/GetStream/mongodb-activity-feed
Comparison of MongoDB Activity Feed performance vs Stream:


TL/DR the repo is functional and tested. Stream, Redis or Cassandra are
better options for building an activity feed.
1 of 11

Recommended

QuerySurge - the automated Data Testing solution by
QuerySurge - the automated Data Testing solutionQuerySurge - the automated Data Testing solution
QuerySurge - the automated Data Testing solutionRTTS
50K views28 slides
AWS DynamoDB by
AWS DynamoDBAWS DynamoDB
AWS DynamoDBSuman Debnath
181 views14 slides
Presentation on Database management system by
Presentation on Database management systemPresentation on Database management system
Presentation on Database management systemPrerana Bhattarai
37.6K views39 slides
9. Document Oriented Databases by
9. Document Oriented Databases9. Document Oriented Databases
9. Document Oriented DatabasesFabio Fumarola
19.7K views62 slides
How MongoDB can accelerate a path to GDPR compliance by
How MongoDB can accelerate a path to GDPR complianceHow MongoDB can accelerate a path to GDPR compliance
How MongoDB can accelerate a path to GDPR complianceMongoDB
2.4K views53 slides
Practical Object Oriented Models In Sql by
Practical Object Oriented Models In SqlPractical Object Oriented Models In Sql
Practical Object Oriented Models In SqlKarwin Software Solutions LLC
90.6K views82 slides

More Related Content

What's hot

Algebra relacional by
Algebra relacionalAlgebra relacional
Algebra relacionalLuis Polanco-Balcazar
571 views73 slides
Indexing with MongoDB by
Indexing with MongoDBIndexing with MongoDB
Indexing with MongoDBMongoDB
12.6K views24 slides
Optimizing Cypher Queries in Neo4j by
Optimizing Cypher Queries in Neo4jOptimizing Cypher Queries in Neo4j
Optimizing Cypher Queries in Neo4jNeo4j
10.7K views71 slides
Summary data visualization by
Summary data visualizationSummary data visualization
Summary data visualizationNovita Sari
92 views10 slides
Image retrieval by
Image retrievalImage retrieval
Image retrievalSujata Regoti
765 views15 slides

What's hot(20)

Indexing with MongoDB by MongoDB
Indexing with MongoDBIndexing with MongoDB
Indexing with MongoDB
MongoDB12.6K views
Optimizing Cypher Queries in Neo4j by Neo4j
Optimizing Cypher Queries in Neo4jOptimizing Cypher Queries in Neo4j
Optimizing Cypher Queries in Neo4j
Neo4j10.7K views
Summary data visualization by Novita Sari
Summary data visualizationSummary data visualization
Summary data visualization
Novita Sari92 views
Apache Spark - Basics of RDD | Big Data Hadoop Spark Tutorial | CloudxLab by CloudxLab
Apache Spark - Basics of RDD | Big Data Hadoop Spark Tutorial | CloudxLabApache Spark - Basics of RDD | Big Data Hadoop Spark Tutorial | CloudxLab
Apache Spark - Basics of RDD | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab1.9K views
Microsoft Azure Big Data Analytics by Mark Kromer
Microsoft Azure Big Data AnalyticsMicrosoft Azure Big Data Analytics
Microsoft Azure Big Data Analytics
Mark Kromer4.2K views
Database Anti Patterns by Robert Treat
Database Anti PatternsDatabase Anti Patterns
Database Anti Patterns
Robert Treat10.1K views
OrientDB by aemadrid
OrientDBOrientDB
OrientDB
aemadrid2.2K views
DBMS Integrity rule by GirdharRatne
DBMS Integrity ruleDBMS Integrity rule
DBMS Integrity rule
GirdharRatne3.5K views
Apache Pig: A big data processor by Tushar B Kute
Apache Pig: A big data processorApache Pig: A big data processor
Apache Pig: A big data processor
Tushar B Kute3.2K views
Deep Learning for Domain-Specific Entity Extraction from Unstructured Text wi... by Databricks
Deep Learning for Domain-Specific Entity Extraction from Unstructured Text wi...Deep Learning for Domain-Specific Entity Extraction from Unstructured Text wi...
Deep Learning for Domain-Specific Entity Extraction from Unstructured Text wi...
Databricks2.4K views
Solr Graph Query: Presented by Kevin Watters, KMW Technology by Lucidworks
Solr Graph Query: Presented by Kevin Watters, KMW TechnologySolr Graph Query: Presented by Kevin Watters, KMW Technology
Solr Graph Query: Presented by Kevin Watters, KMW Technology
Lucidworks4.6K views
Trees In The Database - Advanced data structures by Lorenzo Alberton
Trees In The Database - Advanced data structuresTrees In The Database - Advanced data structures
Trees In The Database - Advanced data structures
Lorenzo Alberton144.6K views

Similar to Scalable News Feed with Mongo DB

LatJUG. Google App Engine by
LatJUG. Google App EngineLatJUG. Google App Engine
LatJUG. Google App Enginedenis Udod
526 views40 slides
SEO for Large/Enterprise Websites - Data & Tech Side by
SEO for Large/Enterprise Websites - Data & Tech SideSEO for Large/Enterprise Websites - Data & Tech Side
SEO for Large/Enterprise Websites - Data & Tech SideDominic Woodman
3.2K views148 slides
KWizCom Aggregation solutions for sharepoint by
KWizCom Aggregation solutions for sharepointKWizCom Aggregation solutions for sharepoint
KWizCom Aggregation solutions for sharepointNimrod Geva
674 views40 slides
MongoDB.local Dallas 2019: MongoDB Stitch Tutorial by
MongoDB.local Dallas 2019: MongoDB Stitch TutorialMongoDB.local Dallas 2019: MongoDB Stitch Tutorial
MongoDB.local Dallas 2019: MongoDB Stitch TutorialMongoDB
234 views38 slides
Log analysis and pro use cases for search marketers online version (1) by
Log analysis and pro use cases for search marketers online version (1)Log analysis and pro use cases for search marketers online version (1)
Log analysis and pro use cases for search marketers online version (1)David Sottimano
2.7K views61 slides
Serverless Analytics and Monitoring For Your Cloud App by
Serverless Analytics and Monitoring For Your Cloud AppServerless Analytics and Monitoring For Your Cloud App
Serverless Analytics and Monitoring For Your Cloud AppAtlassian
14.5K views122 slides

Similar to Scalable News Feed with Mongo DB(20)

LatJUG. Google App Engine by denis Udod
LatJUG. Google App EngineLatJUG. Google App Engine
LatJUG. Google App Engine
denis Udod526 views
SEO for Large/Enterprise Websites - Data & Tech Side by Dominic Woodman
SEO for Large/Enterprise Websites - Data & Tech SideSEO for Large/Enterprise Websites - Data & Tech Side
SEO for Large/Enterprise Websites - Data & Tech Side
Dominic Woodman3.2K views
KWizCom Aggregation solutions for sharepoint by Nimrod Geva
KWizCom Aggregation solutions for sharepointKWizCom Aggregation solutions for sharepoint
KWizCom Aggregation solutions for sharepoint
Nimrod Geva674 views
MongoDB.local Dallas 2019: MongoDB Stitch Tutorial by MongoDB
MongoDB.local Dallas 2019: MongoDB Stitch TutorialMongoDB.local Dallas 2019: MongoDB Stitch Tutorial
MongoDB.local Dallas 2019: MongoDB Stitch Tutorial
MongoDB234 views
Log analysis and pro use cases for search marketers online version (1) by David Sottimano
Log analysis and pro use cases for search marketers online version (1)Log analysis and pro use cases for search marketers online version (1)
Log analysis and pro use cases for search marketers online version (1)
David Sottimano2.7K views
Serverless Analytics and Monitoring For Your Cloud App by Atlassian
Serverless Analytics and Monitoring For Your Cloud AppServerless Analytics and Monitoring For Your Cloud App
Serverless Analytics and Monitoring For Your Cloud App
Atlassian14.5K views
MongoDB.local Seattle 2019: MongoDB Stitch Tutorial by MongoDB
MongoDB.local Seattle 2019: MongoDB Stitch TutorialMongoDB.local Seattle 2019: MongoDB Stitch Tutorial
MongoDB.local Seattle 2019: MongoDB Stitch Tutorial
MongoDB416 views
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th... by MongoDB
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB World 2018: Ch-Ch-Ch-Ch-Changes: Taking Your Stitch Application to th...
MongoDB339 views
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB by MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDBMongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB .local Houston 2019: Wide Ranging Analytical Solutions on MongoDB
MongoDB209 views
Lizards & Pumpkins Catalog Replacement at mm17de by vinaikopp
Lizards & Pumpkins Catalog Replacement at mm17deLizards & Pumpkins Catalog Replacement at mm17de
Lizards & Pumpkins Catalog Replacement at mm17de
vinaikopp1K views
MongoDB.local Atlanta: MongoDB Stitch Tutorial by MongoDB
MongoDB.local Atlanta: MongoDB Stitch TutorialMongoDB.local Atlanta: MongoDB Stitch Tutorial
MongoDB.local Atlanta: MongoDB Stitch Tutorial
MongoDB229 views
Angular JS - Introduction by Sagar Acharya
Angular JS - IntroductionAngular JS - Introduction
Angular JS - Introduction
Sagar Acharya2.4K views
Improving Drupal Performances by Vladimir Ilic
Improving Drupal PerformancesImproving Drupal Performances
Improving Drupal Performances
Vladimir Ilic5.4K views
Large Data Volume Salesforce experiences by Cidar Mendizabal
Large Data Volume Salesforce experiencesLarge Data Volume Salesforce experiences
Large Data Volume Salesforce experiences
Cidar Mendizabal2.5K views
Making operations visible - devopsdays tokyo 2013 by Nick Galbreath
Making operations visible  - devopsdays tokyo 2013Making operations visible  - devopsdays tokyo 2013
Making operations visible - devopsdays tokyo 2013
Nick Galbreath13.8K views
Making operations visible - Nick Gallbreath by Devopsdays
Making operations visible - Nick GallbreathMaking operations visible - Nick Gallbreath
Making operations visible - Nick Gallbreath
Devopsdays1.5K views
Partner Webinar: The Scaling Checklist for MongoDB - 100GB and beyond by MongoDB
Partner Webinar: The Scaling Checklist for MongoDB - 100GB and beyondPartner Webinar: The Scaling Checklist for MongoDB - 100GB and beyond
Partner Webinar: The Scaling Checklist for MongoDB - 100GB and beyond
MongoDB14.3K views

More from Thierry Schellenbach

13 tips for a highly engaging feed by
13 tips for a highly engaging feed13 tips for a highly engaging feed
13 tips for a highly engaging feedThierry Schellenbach
20.6K views16 slides
AWS Summit Amsterdam - Thierry Schellenbach Founder/ Fashiolista by
AWS Summit Amsterdam - Thierry Schellenbach Founder/ FashiolistaAWS Summit Amsterdam - Thierry Schellenbach Founder/ Fashiolista
AWS Summit Amsterdam - Thierry Schellenbach Founder/ FashiolistaThierry Schellenbach
834 views15 slides
Feedly & Cassandra at Fashiolista by
Feedly & Cassandra at FashiolistaFeedly & Cassandra at Fashiolista
Feedly & Cassandra at FashiolistaThierry Schellenbach
2.8K views31 slides
Fashiolista Notifications by
Fashiolista NotificationsFashiolista Notifications
Fashiolista NotificationsThierry Schellenbach
679 views12 slides
Performance metrics for a social network by
Performance metrics for a social networkPerformance metrics for a social network
Performance metrics for a social networkThierry Schellenbach
18.7K views38 slides
Just a millisecond_ by
Just a millisecond_Just a millisecond_
Just a millisecond_Thierry Schellenbach
610 views13 slides

More from Thierry Schellenbach(6)

Recently uploaded

What is Whirling Hygrometer.pdf by
What is Whirling Hygrometer.pdfWhat is Whirling Hygrometer.pdf
What is Whirling Hygrometer.pdfIIT KHARAGPUR
11 views3 slides
Effect of deep chemical mixing columns on properties of surrounding soft clay... by
Effect of deep chemical mixing columns on properties of surrounding soft clay...Effect of deep chemical mixing columns on properties of surrounding soft clay...
Effect of deep chemical mixing columns on properties of surrounding soft clay...AltinKaradagli
6 views10 slides
Dynamics of Hard-Magnetic Soft Materials by
Dynamics of Hard-Magnetic Soft MaterialsDynamics of Hard-Magnetic Soft Materials
Dynamics of Hard-Magnetic Soft MaterialsShivendra Nandan
14 views32 slides
START Newsletter 3 by
START Newsletter 3START Newsletter 3
START Newsletter 3Start Project
5 views25 slides
A multi-microcontroller-based hardware for deploying Tiny machine learning mo... by
A multi-microcontroller-based hardware for deploying Tiny machine learning mo...A multi-microcontroller-based hardware for deploying Tiny machine learning mo...
A multi-microcontroller-based hardware for deploying Tiny machine learning mo...IJECEIAES
13 views10 slides
Control Systems Feedback.pdf by
Control Systems Feedback.pdfControl Systems Feedback.pdf
Control Systems Feedback.pdfLGGaming5
5 views39 slides

Recently uploaded(20)

What is Whirling Hygrometer.pdf by IIT KHARAGPUR
What is Whirling Hygrometer.pdfWhat is Whirling Hygrometer.pdf
What is Whirling Hygrometer.pdf
IIT KHARAGPUR 11 views
Effect of deep chemical mixing columns on properties of surrounding soft clay... by AltinKaradagli
Effect of deep chemical mixing columns on properties of surrounding soft clay...Effect of deep chemical mixing columns on properties of surrounding soft clay...
Effect of deep chemical mixing columns on properties of surrounding soft clay...
AltinKaradagli6 views
Dynamics of Hard-Magnetic Soft Materials by Shivendra Nandan
Dynamics of Hard-Magnetic Soft MaterialsDynamics of Hard-Magnetic Soft Materials
Dynamics of Hard-Magnetic Soft Materials
Shivendra Nandan14 views
A multi-microcontroller-based hardware for deploying Tiny machine learning mo... by IJECEIAES
A multi-microcontroller-based hardware for deploying Tiny machine learning mo...A multi-microcontroller-based hardware for deploying Tiny machine learning mo...
A multi-microcontroller-based hardware for deploying Tiny machine learning mo...
IJECEIAES13 views
Control Systems Feedback.pdf by LGGaming5
Control Systems Feedback.pdfControl Systems Feedback.pdf
Control Systems Feedback.pdf
LGGaming55 views
_MAKRIADI-FOTEINI_diploma thesis.pptx by fotinimakriadi
_MAKRIADI-FOTEINI_diploma thesis.pptx_MAKRIADI-FOTEINI_diploma thesis.pptx
_MAKRIADI-FOTEINI_diploma thesis.pptx
fotinimakriadi7 views
Design of machine elements-UNIT 3.pptx by gopinathcreddy
Design of machine elements-UNIT 3.pptxDesign of machine elements-UNIT 3.pptx
Design of machine elements-UNIT 3.pptx
gopinathcreddy29 views
MSA Website Slideshow (16).pdf by msaucla
MSA Website Slideshow (16).pdfMSA Website Slideshow (16).pdf
MSA Website Slideshow (16).pdf
msaucla58 views
Literature review and Case study on Commercial Complex in Nepal, Durbar mall,... by AakashShakya12
Literature review and Case study on Commercial Complex in Nepal, Durbar mall,...Literature review and Case study on Commercial Complex in Nepal, Durbar mall,...
Literature review and Case study on Commercial Complex in Nepal, Durbar mall,...
AakashShakya1263 views
zincalume water storage tank design.pdf by 3D LABS
zincalume water storage tank design.pdfzincalume water storage tank design.pdf
zincalume water storage tank design.pdf
3D LABS5 views
SUMIT SQL PROJECT SUPERSTORE 1.pptx by Sumit Jadhav
SUMIT SQL PROJECT SUPERSTORE 1.pptxSUMIT SQL PROJECT SUPERSTORE 1.pptx
SUMIT SQL PROJECT SUPERSTORE 1.pptx
Sumit Jadhav 11 views
Thermal aware task assignment for multicore processors using genetic algorithm by IJECEIAES
Thermal aware task assignment for multicore processors using genetic algorithm Thermal aware task assignment for multicore processors using genetic algorithm
Thermal aware task assignment for multicore processors using genetic algorithm
IJECEIAES31 views
Update 42 models(Diode/General ) in SPICE PARK(DEC2023) by Tsuyoshi Horigome
Update 42 models(Diode/General ) in SPICE PARK(DEC2023)Update 42 models(Diode/General ) in SPICE PARK(DEC2023)
Update 42 models(Diode/General ) in SPICE PARK(DEC2023)

Scalable News Feed with Mongo DB

  • 2. Feeds: 
 
 Follow anything (users, sites, stocks, bands, companies, shops, etc.,) What is a feed?
  • 3. MongoDB Activity Feed Activity Feeds Created this while working on a blogpost https://github.com/GetStream/mongodb-activity-feed
  • 5. The API - Timeline Feed Activity Feeds
  • 6. The API - Notification Feed Activity Feeds
  • 7. The API - Ranking Methods Activity Feeds
  • 8. Architecture Activity Feeds • Unordered Bulk Writes • Conflict-free replicated Data Types • Bull + Redis for Queuing
  • 9. CRDTs Activity Feeds You might be wondering why we’re using an operation log instead of just storing the activities. If you simply add and remove activities you run into problems with locking. IE. a user likes something, unlikes it and likes it again. If you’re just adding and removing activities you end up with either: • Incorrect state (Something shows as liked when you unliked it)
 • Locks (i.e. which reduce the write capacity of your system)
 The best way to solve this is using CRDTs. Instead of adding or removing the activity you store that the activity should be added/removed together with the time of the operation. This approach prevents race conditions and doesn't require locking.
  • 10. Schema Activity Feeds • Activity: The actual data of the activity
 • Activity Feed: An operation log. This stores that an activity should be added to a certain feed.
 • Feed Group: A group of feeds, for example: user, notification, timeline etc.
 • Feed: A specific feed, ie user:scott, notification:josh, timeline:tom, etc. • Follow: A follow relationships between 2 feeds.
  • 11. MongoDB Activity Feed Activity Feeds Repo: https://github.com/GetStream/mongodb-activity-feed Comparison of MongoDB Activity Feed performance vs Stream: 
 TL/DR the repo is functional and tested. Stream, Redis or Cassandra are better options for building an activity feed.