SlideShare a Scribd company logo
Scaling the World’s Largest Django App

Jason Yan                 David Cramer
@jasonyan                    @zeeg
What is DISQUS?
What is DISQUS?


            dis·cuss • dĭ-skŭs'

We are a comment system with an emphasis on
           connecting communities




              http://disqus.com/about/
What is Scale?

                     Number of Visitors
300M
250M
200M
150M
100M
 50M



Our traffic at a glance
17,000 requests/second peak
450,000 websites
15 million profiles
75 million comments
250 million visitors (August 2010)
Our Challenges


• We can’t predict when things will happen
  • Random celebrity gossip
  • Natural disasters
• Discussions never expire
  • We can’t keep those millions of articles from
    2008 in the cache
  • You don’t know in advance (generally) where the
    traffic will be
  • Especially with dynamic paging, realtime, sorting,
    personal prefs, etc.
Our Challenges (cont’d)


• High availability
  • Not a destination site
  • Difficult to schedule maintenance
Server Architecture
Server Architecture - Load Balancing
• Load Balancing                          • High Availability
  • Software, HAProxy                       • heartbeat
     • High performance, intelligent
       server availability checking
     • Bonus: Nice statistics reporting




                                                     Image Source: http://haproxy.1wt.eu/
Server Architecture



• ~100 Servers
 • 30% Web Servers (Apache + mod_wsgi)
 • 10% Databases (PostgreSQL)
 • 25% Cache Servers (memcached)
 • 20% Load Balancing / High Availability
   (HAProxy + heartbeat)
 • 15% Utility Servers (Python scripts)
Server Architecture - Web Servers


• Apache 2.2
• mod_wsgi
  • Using `maximum-requests` to
    plug memory leaks.

• Performance Monitoring
  • Custom middleware
    (PerformanceLogMiddleware)
  • Ships performance statistics
    (DB queries, external calls,
    template rendering, etc) through
    syslog
  • Collected and graphed through
    Ganglia
Server Architecture - Database




• PostgreSQL
• Slony-I for Replication
  • Trigger-based
  • Read slaves for extra read capacity
  • Failover master database for high
    availability
Server Architecture - Database

• Make sure indexes fit in memory and
  measure I/O
 • High I/O generally means slow queries
   due to missing indexes or indexes not in
   buffer cache
• Log Slow Queries
 • syslog-ng + pgFouine + cron to automate
   slow query logging
Server Architecture - Database



• Use connection pooling
 • Django doesn’t do this for you
 • We use pgbouncer
 • Limits the maximum number of
   connections your database needs to
   handle
 • Save on costly opening and tearing down
   of new database connections
Our Data Model
Partitioning




• Fairly easy to implement, quick wins
• Done at the application level
  • Data is replayed by Slony
• Two methods of data separation
Vertical Partitioning
Vertical partitioning involves creating tables with fewer columns
  and using additional tables to store the remaining columns.



     Forums         Posts             Users         Sentry




          http://en.wikipedia.org/wiki/Partition_(database)
Pythonic Joins


            Allows us to separate datasets

posts = Post.objects.all()[0:25]

# store users in a dictionary based on primary key
users = dict(
    (u.pk, u) for u in 
    User.objects.filter(pk__in=set(p.user_id for p in posts))
)

# map users to their posts
for p in posts:
  p._user_cache = users.get(p.user_id)
Pythonic Joins (cont’d)



• Slower than at database level
    • But not enough that you should care
    • Trading performance for scale
• Allows us to separate data
    • Easy vertical partitioning
• More efficient caching
    • get_many, object-per-row cache
Designating Masters




• Alleviates some of the write load on your
  primary application master
• Masters exist under specific conditions:
  • application use case
  • partitioned data
• Database routers make this (fairly) easy
Routing by Application




class ApplicationRouter(object):
    def db_for_read(self, model, **hints):
        instance = hints.get('instance')
        if not instance:
            return None

        app_label = instance._meta.app_label

        return get_application_alias(app_label)
Horizontal Partitioning
Horizontal partitioning (also known as sharding) involves splitting
               one set of data into different tables.



      Disqus      Your Blog            CNN        Telegraph




           http://en.wikipedia.org/wiki/Partition_(database)
Horizontal Partitions




• Some forums have very large datasets
• Partners need high availability
• Helps scale the write load on the master
• We rely more on vertical partitions
Routing by Partition

class ForumPartitionRouter(object):
    def db_for_read(self, model, **hints):
        instance = hints.get('instance')
        if not instance:
            return None

        forum_id = getattr(instance, 'forum_id', None)
        if not forum_id:
              return None

        return get_forum_alias(forum_id)


# What we used to do
Post.objects.filter(forum=forum)


# Now, making sure hints are available
forum.post_set.all()
Optimizing QuerySets




• We really dislike raw SQL
  • It creates more work when dealing with
    partitions
• Built-in cache allows sub-slicing
  • But isn’t always needed
  • We removed this cache
Removing the Cache


• Django internally caches the results of your QuerySet
  • This adds additional memory overhead

     # 1 query
     qs = Model.objects.all()[0:100]

     # 0 queries (we don’t need this behavior)
     qs = qs[0:10]

     # 1 query
     qs = qs.filter(foo=bar)


• Many times you only need to view a result set once
• So we built SkinnyQuerySet
Removing the Cache (cont’d)

Optimizing memory usage by removing the cache
 class SkinnyQuerySet(QuerySet):
     def __iter__(self):
         if self._result_cache is not None:
             # __len__ must have been run
             return iter(self._result_cache)

        has_run = getattr(self, 'has_run', False)
        if has_run:
            raise QuerySetDoubleIteration("...")
        self.has_run = True
        # We wanted .iterator() as the default
        return self.iterator()



                http://gist.github.com/550438
Atomic Updates




• Keeps your data consistent
• save() isnt thread-safe
  • use update() instead
• Great for things like counters
  • But should be considered for all write
    operations
Atomic Updates (cont’d)


  Thread safety is impossible with .save()
Request 1

post = Post(pk=1)
# a moderator approves
post.approved = True
post.save()

Request 2

post = Post(pk=1)
# the author adjusts their message
post.message = ‘Hello!’
post.save()
Atomic Updates (cont’d)


            So we need atomic updates
Request 1

post = Post(pk=1)
# a moderator approves
Post.objects.filter(pk=post.pk)
            .update(approved=True)

Request 2

post = Post(pk=1)
# the author adjusts their message
Post.objects.filter(pk=post.pk)
            .update(message=‘Hello!’)
Atomic Updates (cont’d)


           A better way to approach updates
def update(obj, using=None, **kwargs):
    """
    Updates specified attributes on the current instance.
    """
    assert obj, "Instance has not yet been created."
    obj.__class__._base_manager.using(using)
                                .filter(pk=obj)
                                .update(**kwargs)
    for k, v in kwargs.iteritems():
        if isinstance(v, ExpressionNode):
            # NotImplemented
            continue
        setattr(obj, k, v)



http://github.com/andymccurdy/django-tips-and-tricks/blob/master/model_update.py
Delayed Signals




• Queueing low priority tasks
 • even if they’re fast
• Asynchronous (Delayed) signals
 • very friendly to the developer
 • ..but not as friendly as real signals
Delayed Signals (cont’d)



  We send a specific serialized version
   of the model for delayed signals

from disqus.common.signals import delayed_save

def my_func(data, sender, created, **kwargs):
    print data[‘id’]

delayed_save.connect(my_func, sender=Post)




 This is all handled through our Queue
Caching




• Memcached
• Use pylibmc (newer libMemcached-based)
 • Ticket #11675 (add pylibmc support)
 • Third party applications:
   • django-newcache, django-pylibmc
Caching (cont’d)



• libMemcached / pylibmc is configurable with
  “behaviors”.
• Memcached “single point of failure”
  • Distributed system, but we must take
    precautions.
  • Connection timeout to memcached can stall
    requests.
    • Use `_auto_eject_hosts` and
      `_retry_timeout` behaviors to prevent
      reconnecting to dead caches.
Caching (cont’d)



   • Default (naive) hashing behavior
     • Modulo hashed cache key cache for index
       to server list.
     • Removal of a server causes majority of
       cache keys to be remapped to new
       servers.

CACHE_SERVERS = [‘10.0.0.1’, ‘10.0.0.2’]
key = ‘my_cache_key’
cache_server = CACHE_SERVERS[hash(key) % len(CACHE_SERVERS)]
Caching (cont’d)

• Better approach: consistent hashing
  • libMemcached (pylibmc) uses libketama
    (http://tinyurl.com/lastfm-libketama)


  • Addition / removal of a cache server
    remaps (K/n) cache keys
    (where K=number of keys and n=number of servers)




                 Image Source: http://sourceforge.net/apps/mediawiki/kai/index.php?title=Introduction
Caching (cont’d)


• Thundering herd (stampede) problem
  • Invalidating a heavily accessed cache key causes many
    clients to refill cache.
  • But everyone refetching to fill the cache from the data
    store or reprocessing data can cause things to get even
    slower.
  • Most times, it’s ideal to return the previously invalidated
    cache value and let a single client refill the cache.
  • django-newcache or MintCache (http://
    djangosnippets.org/snippets/793/) will do this for you.
  • Prefer filling cache on invalidation instead of deleting
    from cache also helps to prevent the thundering herd
    problem.
Transactions


• TransactionMiddleware got us started, but
  down the road became a burden
• For postgresql_psycopg2, there’s a database
  option, OPTIONS[‘autocommit’]
  • Each query is in its own transaction. This
    means each request won’t start in a
    transaction.
    • But sometimes we want transactions
      (e.g., saving multiple objects and rolling
      back on error)
Transactions (cont’d)


• Tips:
  • Use autocommit for read slave databases.
  • Isolate slow functions (e.g., external calls,
    template rendering) from transactions.
  • Selective autocommit
    • Most read-only views don’t need to be
      in transactions.
    • Start in autocommit and switch to a
      transaction on write.
Scaling the Team




• Small team of engineers
• Monthly users / developers = 40m
• Which means writing tests..
• ..and having a dead simple workflow
Keeping it Simple




• A developer can be up and running in a few
  minutes
 • assuming postgres and other server
   applications are already installed
 • pip, virtualenv
 • settings.py
Setting Up Local




1. createdb -E UTF-8 disqus
2. git clone git://repo
3. mkvirtualenv disqus
4. pip install -U -r requirements.txt
5. ./manage.py syncdb && ./manage.py migrate
Sane Defaults


settings.py
from disqus.conf.settings.default import *

try:
    from local_settings import *
except ImportError:
    import sys, traceback
    sys.stderr.write("Can't find 'localsettings.py’n”)
    sys.stderr.write("nThe exception was:nn")
    traceback.print_exc()



local_settings.py
from disqus.conf.settings.dev import *
Continuous Integration



• Daily deploys with Fabric
  • several times an hour on some days
• Hudson keeps our builds going
  • combined with Selenium
• Post-commit hooks for quick testing
  • like Pyflakes
• Reverting to a previous version is a matter of
  seconds
Continuous Integration (cont’d)

 Hudson makes integration easy
Testing



• It’s not fun breaking things when you’re the new
  guy
• Our testing process is fairly heavy
• 70k (Python) LOC, 73% coverage, 20 min suite
• Custom Test Runner (unittest)
  • We needed XML, Selenium, Query Counts
  • Database proxies (for read-slave testing)
  • Integration with our Queue
Testing (cont’d)


Query Counts
# failures yield a dump of queries
def test_read_slave(self):
    Model.objects.using(‘read_slave’).count()
    self.assertQueryCount(1, ‘read_slave’)


Selenium
def test_button(self):
    self.selenium.click('//a[@class=”dsq-button”]')



Queue Integration
class WorkerTest(DisqusTest):
    workers = [‘fire_signal’]

    def test_delayed_signal(self):
        ...
Bug Tracking



• Switched from Trac to Redmine
  • We wanted Subtasks
• Emailing exceptions is a bad idea
  • Even if its localhost
• Previously using django-db-log to aggregate
  errors to a single point
• We’ve overhauled db log and are releasing
  Sentry
django-sentry

Groups messages intelligently




   http://github.com/dcramer/django-sentry
django-sentry (cont’d)

Similar feel to Django’s debugger




    http://github.com/dcramer/django-sentry
Feature Switches



• We needed a safety in case a feature wasn’t
  performing well at peak
  • it had to respond without delay, globally,
    and without writing to disk
• Allows us to work out of trunk (mostly)
• Easy to release new features to a portion of
  your audience
• Also nice for “Labs” type projects
Feature Switches (cont’d)
Final Thoughts


• The language (usually) isn’t your problem
• We like Django
  • But we maintain local patches
• Some tickets don’t have enough of a following
  • Patches, like #17, completely change
    Django..
  • ..arguably in a good way
• Others don’t have champions
      Ticket #17 describes making the ORM an identify mapper
Housekeeping




       Birds of a Feather
   Want to learn from others about
  performance and scaling problems?
           Or play some StarCraft 2?


          We’re Hiring!

DISQUS is looking for amazing engineers
Questions
References


django-sentry
http://github.com/dcramer/django-sentry

Our Feature Switches
http://cl.ly/2FYt

Andy McCurdy’s update()
http://github.com/andymccurdy/django-tips-and-tricks

Our PyFlakes Fork
http://github.com/dcramer/pyflakes

SkinnyQuerySet
http://gist.github.com/550438

django-newcache
http://github.com/ericflo/django-newcache

attach_foreignkey (Pythonic Joins)
http://gist.github.com/567356

More Related Content

What's hot

Cours javascript
Cours javascriptCours javascript
Cours javascript
krymo
 
Rapport DVWA: File Upload
Rapport DVWA: File UploadRapport DVWA: File Upload
Rapport DVWA: File Upload
Ayoub Rouzi
 
Gitlab Training with GIT and SourceTree
Gitlab Training with GIT and SourceTreeGitlab Training with GIT and SourceTree
Gitlab Training with GIT and SourceTree
Teerapat Khunpech
 
Twitter의 snowflake 소개 및 활용
Twitter의 snowflake 소개 및 활용Twitter의 snowflake 소개 및 활용
Twitter의 snowflake 소개 및 활용
흥배 최
 
Une introduction à Hive
Une introduction à HiveUne introduction à Hive
Une introduction à Hive
Modern Data Stack France
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
Jo Hoon
 
Java 8 - interfaces
Java 8 - interfacesJava 8 - interfaces
Java 8 - interfaces
Franck SIMON
 
Support JEE Servlet Jsp MVC M.Youssfi
Support JEE Servlet Jsp MVC M.YoussfiSupport JEE Servlet Jsp MVC M.Youssfi
Support JEE Servlet Jsp MVC M.Youssfi
ENSET, Université Hassan II Casablanca
 
Tp2 - WS avec JAXRS
Tp2 - WS avec JAXRSTp2 - WS avec JAXRS
Tp2 - WS avec JAXRS
Lilia Sfaxi
 
Rapport DVWA: CSRF
Rapport DVWA: CSRFRapport DVWA: CSRF
Rapport DVWA: CSRF
Ayoub Rouzi
 
Cours design pattern m youssfi partie 3 decorateur
Cours design pattern m youssfi partie 3 decorateurCours design pattern m youssfi partie 3 decorateur
Cours design pattern m youssfi partie 3 decorateur
ENSET, Université Hassan II Casablanca
 
Support de Cours JSF2 Première partie Intégration avec Spring
Support de Cours JSF2 Première partie Intégration avec SpringSupport de Cours JSF2 Première partie Intégration avec Spring
Support de Cours JSF2 Première partie Intégration avec Spring
ENSET, Université Hassan II Casablanca
 
코틀린 멀티플랫폼, 미지와의 조우
코틀린 멀티플랫폼, 미지와의 조우코틀린 멀티플랫폼, 미지와의 조우
코틀린 멀티플랫폼, 미지와의 조우
Arawn Park
 
Basic and Advanced Analysis of Ceph Volume Backend Driver in Cinder - John Haan
Basic and Advanced Analysis of Ceph Volume Backend Driver in Cinder - John HaanBasic and Advanced Analysis of Ceph Volume Backend Driver in Cinder - John Haan
Basic and Advanced Analysis of Ceph Volume Backend Driver in Cinder - John Haan
Ceph Community
 
Intégration continue et déploiement continue avec Jenkins
Intégration continue et déploiement continue avec JenkinsIntégration continue et déploiement continue avec Jenkins
Intégration continue et déploiement continue avec Jenkins
Kokou Gaglo
 
Introduction à spring boot
Introduction à spring bootIntroduction à spring boot
Introduction à spring boot
Antoine Rey
 
Apache Spark : Genel Bir Bakış
Apache Spark : Genel Bir BakışApache Spark : Genel Bir Bakış
Apache Spark : Genel Bir Bakış
Burak KÖSE
 
Background processing with hangfire
Background processing with hangfireBackground processing with hangfire
Background processing with hangfire
Aleksandar Bozinovski
 
Support NodeJS avec TypeScript Express MongoDB
Support NodeJS avec TypeScript Express MongoDBSupport NodeJS avec TypeScript Express MongoDB
Support NodeJS avec TypeScript Express MongoDB
ENSET, Université Hassan II Casablanca
 
Support JEE Spring Inversion de Controle IOC et Spring MVC
Support JEE Spring Inversion de Controle IOC et Spring MVCSupport JEE Spring Inversion de Controle IOC et Spring MVC
Support JEE Spring Inversion de Controle IOC et Spring MVC
ENSET, Université Hassan II Casablanca
 

What's hot (20)

Cours javascript
Cours javascriptCours javascript
Cours javascript
 
Rapport DVWA: File Upload
Rapport DVWA: File UploadRapport DVWA: File Upload
Rapport DVWA: File Upload
 
Gitlab Training with GIT and SourceTree
Gitlab Training with GIT and SourceTreeGitlab Training with GIT and SourceTree
Gitlab Training with GIT and SourceTree
 
Twitter의 snowflake 소개 및 활용
Twitter의 snowflake 소개 및 활용Twitter의 snowflake 소개 및 활용
Twitter의 snowflake 소개 및 활용
 
Une introduction à Hive
Une introduction à HiveUne introduction à Hive
Une introduction à Hive
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
 
Java 8 - interfaces
Java 8 - interfacesJava 8 - interfaces
Java 8 - interfaces
 
Support JEE Servlet Jsp MVC M.Youssfi
Support JEE Servlet Jsp MVC M.YoussfiSupport JEE Servlet Jsp MVC M.Youssfi
Support JEE Servlet Jsp MVC M.Youssfi
 
Tp2 - WS avec JAXRS
Tp2 - WS avec JAXRSTp2 - WS avec JAXRS
Tp2 - WS avec JAXRS
 
Rapport DVWA: CSRF
Rapport DVWA: CSRFRapport DVWA: CSRF
Rapport DVWA: CSRF
 
Cours design pattern m youssfi partie 3 decorateur
Cours design pattern m youssfi partie 3 decorateurCours design pattern m youssfi partie 3 decorateur
Cours design pattern m youssfi partie 3 decorateur
 
Support de Cours JSF2 Première partie Intégration avec Spring
Support de Cours JSF2 Première partie Intégration avec SpringSupport de Cours JSF2 Première partie Intégration avec Spring
Support de Cours JSF2 Première partie Intégration avec Spring
 
코틀린 멀티플랫폼, 미지와의 조우
코틀린 멀티플랫폼, 미지와의 조우코틀린 멀티플랫폼, 미지와의 조우
코틀린 멀티플랫폼, 미지와의 조우
 
Basic and Advanced Analysis of Ceph Volume Backend Driver in Cinder - John Haan
Basic and Advanced Analysis of Ceph Volume Backend Driver in Cinder - John HaanBasic and Advanced Analysis of Ceph Volume Backend Driver in Cinder - John Haan
Basic and Advanced Analysis of Ceph Volume Backend Driver in Cinder - John Haan
 
Intégration continue et déploiement continue avec Jenkins
Intégration continue et déploiement continue avec JenkinsIntégration continue et déploiement continue avec Jenkins
Intégration continue et déploiement continue avec Jenkins
 
Introduction à spring boot
Introduction à spring bootIntroduction à spring boot
Introduction à spring boot
 
Apache Spark : Genel Bir Bakış
Apache Spark : Genel Bir BakışApache Spark : Genel Bir Bakış
Apache Spark : Genel Bir Bakış
 
Background processing with hangfire
Background processing with hangfireBackground processing with hangfire
Background processing with hangfire
 
Support NodeJS avec TypeScript Express MongoDB
Support NodeJS avec TypeScript Express MongoDBSupport NodeJS avec TypeScript Express MongoDB
Support NodeJS avec TypeScript Express MongoDB
 
Support JEE Spring Inversion de Controle IOC et Spring MVC
Support JEE Spring Inversion de Controle IOC et Spring MVCSupport JEE Spring Inversion de Controle IOC et Spring MVC
Support JEE Spring Inversion de Controle IOC et Spring MVC
 

Viewers also liked

Physical Security Presentation
Physical Security PresentationPhysical Security Presentation
Physical Security Presentation
Wajahat Rajab
 
Mri brain anatomy Dr Muhammad Bin Zulfiqar
Mri brain anatomy Dr Muhammad Bin ZulfiqarMri brain anatomy Dr Muhammad Bin Zulfiqar
Mri brain anatomy Dr Muhammad Bin Zulfiqar
Dr. Muhammad Bin Zulfiqar
 
Thai tech startup ecosystem report 2017
Thai tech startup ecosystem report 2017Thai tech startup ecosystem report 2017
Thai tech startup ecosystem report 2017
Techsauce Media
 
Engineering Geology
Engineering GeologyEngineering Geology
Engineering Geology
GAURAV. H .TANDON
 
Process sequence of weaving
Process sequence of weavingProcess sequence of weaving
Process sequence of weaving
Md. Mazadul Hasan Shishir
 
The evolution of mobile phones
The evolution of mobile phonesThe evolution of mobile phones
The evolution of mobile phones
Olivia2590
 
Cardiac cycle ppt (2)
Cardiac cycle ppt (2)Cardiac cycle ppt (2)
Cardiac cycle ppt (2)
Gopi Krishna Rayidi
 
BCG Matrix of Engro foods
BCG Matrix of Engro foodsBCG Matrix of Engro foods
BCG Matrix of Engro foods
Mutahir Bilal
 
4. heredity and evolution
4. heredity and evolution4. heredity and evolution
4. heredity and evolution
Abhay Goyal
 
10+ Getting to Know You Activities for Teens & Adults
10+ Getting to Know You Activities for Teens & Adults10+ Getting to Know You Activities for Teens & Adults
10+ Getting to Know You Activities for Teens & Adults
Shelly Sanchez Terrell
 
Tmj anatomy
Tmj anatomyTmj anatomy
Tmj anatomy
Tony Pious
 
Basics of c++ Programming Language
Basics of c++ Programming LanguageBasics of c++ Programming Language
Basics of c++ Programming Language
Ahmad Idrees
 
How Obama Won Using Digital and Social Media
How Obama Won Using Digital and Social MediaHow Obama Won Using Digital and Social Media
How Obama Won Using Digital and Social Media
James Burnes
 
Micro Expressions
Micro ExpressionsMicro Expressions
Micro Expressions
Yang Ao Wei 楊翱維
 
BUSINESS QUIZ -Round 1
 BUSINESS QUIZ -Round 1 BUSINESS QUIZ -Round 1
BUSINESS QUIZ -Round 1
pradeep acharya
 
Fmcg training modules-bfg
Fmcg training modules-bfgFmcg training modules-bfg
Fmcg training modules-bfg
Romy Cagampan
 
New forever clean 9 booklet
New forever clean 9 bookletNew forever clean 9 booklet
New forever clean 9 booklet
Katalin Hidvegi
 
Tweak Your Resume
Tweak Your ResumeTweak Your Resume
Tweak Your Resume
Chiara Ojeda
 
Coca Cola
Coca ColaCoca Cola
Coca Cola
mixas450
 
Understanding text-structure-powerpoint
Understanding text-structure-powerpointUnderstanding text-structure-powerpoint
Understanding text-structure-powerpoint
aelowans
 

Viewers also liked (20)

Physical Security Presentation
Physical Security PresentationPhysical Security Presentation
Physical Security Presentation
 
Mri brain anatomy Dr Muhammad Bin Zulfiqar
Mri brain anatomy Dr Muhammad Bin ZulfiqarMri brain anatomy Dr Muhammad Bin Zulfiqar
Mri brain anatomy Dr Muhammad Bin Zulfiqar
 
Thai tech startup ecosystem report 2017
Thai tech startup ecosystem report 2017Thai tech startup ecosystem report 2017
Thai tech startup ecosystem report 2017
 
Engineering Geology
Engineering GeologyEngineering Geology
Engineering Geology
 
Process sequence of weaving
Process sequence of weavingProcess sequence of weaving
Process sequence of weaving
 
The evolution of mobile phones
The evolution of mobile phonesThe evolution of mobile phones
The evolution of mobile phones
 
Cardiac cycle ppt (2)
Cardiac cycle ppt (2)Cardiac cycle ppt (2)
Cardiac cycle ppt (2)
 
BCG Matrix of Engro foods
BCG Matrix of Engro foodsBCG Matrix of Engro foods
BCG Matrix of Engro foods
 
4. heredity and evolution
4. heredity and evolution4. heredity and evolution
4. heredity and evolution
 
10+ Getting to Know You Activities for Teens & Adults
10+ Getting to Know You Activities for Teens & Adults10+ Getting to Know You Activities for Teens & Adults
10+ Getting to Know You Activities for Teens & Adults
 
Tmj anatomy
Tmj anatomyTmj anatomy
Tmj anatomy
 
Basics of c++ Programming Language
Basics of c++ Programming LanguageBasics of c++ Programming Language
Basics of c++ Programming Language
 
How Obama Won Using Digital and Social Media
How Obama Won Using Digital and Social MediaHow Obama Won Using Digital and Social Media
How Obama Won Using Digital and Social Media
 
Micro Expressions
Micro ExpressionsMicro Expressions
Micro Expressions
 
BUSINESS QUIZ -Round 1
 BUSINESS QUIZ -Round 1 BUSINESS QUIZ -Round 1
BUSINESS QUIZ -Round 1
 
Fmcg training modules-bfg
Fmcg training modules-bfgFmcg training modules-bfg
Fmcg training modules-bfg
 
New forever clean 9 booklet
New forever clean 9 bookletNew forever clean 9 booklet
New forever clean 9 booklet
 
Tweak Your Resume
Tweak Your ResumeTweak Your Resume
Tweak Your Resume
 
Coca Cola
Coca ColaCoca Cola
Coca Cola
 
Understanding text-structure-powerpoint
Understanding text-structure-powerpointUnderstanding text-structure-powerpoint
Understanding text-structure-powerpoint
 

Similar to DjangoCon 2010 Scaling Disqus

Play Framework and Activator
Play Framework and ActivatorPlay Framework and Activator
Play Framework and Activator
Kevin Webber
 
introduction to node.js
introduction to node.jsintroduction to node.js
introduction to node.js
orkaplan
 
Django at Scale
Django at ScaleDjango at Scale
Django at Scale
bretthoerner
 
Bye bye $GLOBALS['TYPO3_DB']
Bye bye $GLOBALS['TYPO3_DB']Bye bye $GLOBALS['TYPO3_DB']
Bye bye $GLOBALS['TYPO3_DB']
Jan Helke
 
Nodejs - Should Ruby Developers Care?
Nodejs - Should Ruby Developers Care?Nodejs - Should Ruby Developers Care?
Nodejs - Should Ruby Developers Care?
Felix Geisendörfer
 
Where Django Caching Bust at the Seams
Where Django Caching Bust at the SeamsWhere Django Caching Bust at the Seams
Where Django Caching Bust at the Seams
Concentric Sky
 
Django Pro ORM
Django Pro ORMDjango Pro ORM
Django Pro ORM
Alex Gaynor
 
Hosting Ruby Web Apps
Hosting Ruby Web AppsHosting Ruby Web Apps
Hosting Ruby Web Apps
Michael Reinsch
 
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander ZaitsevMigration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Altinity Ltd
 
IVS CTO Night And Day 2018 Winter - [re:Cap] Serverless & Mobile
IVS CTO Night And Day 2018 Winter - [re:Cap] Serverless & MobileIVS CTO Night And Day 2018 Winter - [re:Cap] Serverless & Mobile
IVS CTO Night And Day 2018 Winter - [re:Cap] Serverless & Mobile
Amazon Web Services Japan
 
CouchDB for Web Applications - Erlang Factory London 2009
CouchDB for Web Applications - Erlang Factory London 2009CouchDB for Web Applications - Erlang Factory London 2009
CouchDB for Web Applications - Erlang Factory London 2009
Jason Davies
 
Rails 3 (beta) Roundup
Rails 3 (beta) RoundupRails 3 (beta) Roundup
Rails 3 (beta) Roundup
Wayne Carter
 
Our Puppet Story (GUUG FFG 2015)
Our Puppet Story (GUUG FFG 2015)Our Puppet Story (GUUG FFG 2015)
Our Puppet Story (GUUG FFG 2015)
DECK36
 
Architecting for Microservices Part 2
Architecting for Microservices Part 2Architecting for Microservices Part 2
Architecting for Microservices Part 2
Elana Krasner
 
Rails Tips and Best Practices
Rails Tips and Best PracticesRails Tips and Best Practices
Rails Tips and Best Practices
David Keener
 
What's new in JBoss ON 3.2
What's new in JBoss ON 3.2What's new in JBoss ON 3.2
What's new in JBoss ON 3.2
Thomas Segismont
 
How to Contribute to Apache Usergrid
How to Contribute to Apache UsergridHow to Contribute to Apache Usergrid
How to Contribute to Apache Usergrid
David M. Johnson
 
Intro to node and mongodb 1
Intro to node and mongodb   1Intro to node and mongodb   1
Intro to node and mongodb 1
Mohammad Qureshi
 
PostgreSQL Performance Problems: Monitoring and Alerting
PostgreSQL Performance Problems: Monitoring and AlertingPostgreSQL Performance Problems: Monitoring and Alerting
PostgreSQL Performance Problems: Monitoring and Alerting
Grant Fritchey
 
Elements for an iOS Backend
Elements for an iOS BackendElements for an iOS Backend
Elements for an iOS Backend
Laurent Cerveau
 

Similar to DjangoCon 2010 Scaling Disqus (20)

Play Framework and Activator
Play Framework and ActivatorPlay Framework and Activator
Play Framework and Activator
 
introduction to node.js
introduction to node.jsintroduction to node.js
introduction to node.js
 
Django at Scale
Django at ScaleDjango at Scale
Django at Scale
 
Bye bye $GLOBALS['TYPO3_DB']
Bye bye $GLOBALS['TYPO3_DB']Bye bye $GLOBALS['TYPO3_DB']
Bye bye $GLOBALS['TYPO3_DB']
 
Nodejs - Should Ruby Developers Care?
Nodejs - Should Ruby Developers Care?Nodejs - Should Ruby Developers Care?
Nodejs - Should Ruby Developers Care?
 
Where Django Caching Bust at the Seams
Where Django Caching Bust at the SeamsWhere Django Caching Bust at the Seams
Where Django Caching Bust at the Seams
 
Django Pro ORM
Django Pro ORMDjango Pro ORM
Django Pro ORM
 
Hosting Ruby Web Apps
Hosting Ruby Web AppsHosting Ruby Web Apps
Hosting Ruby Web Apps
 
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander ZaitsevMigration to ClickHouse. Practical guide, by Alexander Zaitsev
Migration to ClickHouse. Practical guide, by Alexander Zaitsev
 
IVS CTO Night And Day 2018 Winter - [re:Cap] Serverless & Mobile
IVS CTO Night And Day 2018 Winter - [re:Cap] Serverless & MobileIVS CTO Night And Day 2018 Winter - [re:Cap] Serverless & Mobile
IVS CTO Night And Day 2018 Winter - [re:Cap] Serverless & Mobile
 
CouchDB for Web Applications - Erlang Factory London 2009
CouchDB for Web Applications - Erlang Factory London 2009CouchDB for Web Applications - Erlang Factory London 2009
CouchDB for Web Applications - Erlang Factory London 2009
 
Rails 3 (beta) Roundup
Rails 3 (beta) RoundupRails 3 (beta) Roundup
Rails 3 (beta) Roundup
 
Our Puppet Story (GUUG FFG 2015)
Our Puppet Story (GUUG FFG 2015)Our Puppet Story (GUUG FFG 2015)
Our Puppet Story (GUUG FFG 2015)
 
Architecting for Microservices Part 2
Architecting for Microservices Part 2Architecting for Microservices Part 2
Architecting for Microservices Part 2
 
Rails Tips and Best Practices
Rails Tips and Best PracticesRails Tips and Best Practices
Rails Tips and Best Practices
 
What's new in JBoss ON 3.2
What's new in JBoss ON 3.2What's new in JBoss ON 3.2
What's new in JBoss ON 3.2
 
How to Contribute to Apache Usergrid
How to Contribute to Apache UsergridHow to Contribute to Apache Usergrid
How to Contribute to Apache Usergrid
 
Intro to node and mongodb 1
Intro to node and mongodb   1Intro to node and mongodb   1
Intro to node and mongodb 1
 
PostgreSQL Performance Problems: Monitoring and Alerting
PostgreSQL Performance Problems: Monitoring and AlertingPostgreSQL Performance Problems: Monitoring and Alerting
PostgreSQL Performance Problems: Monitoring and Alerting
 
Elements for an iOS Backend
Elements for an iOS BackendElements for an iOS Backend
Elements for an iOS Backend
 

More from zeeg

Practicing Continuous Deployment
Practicing Continuous DeploymentPracticing Continuous Deployment
Practicing Continuous Deployment
zeeg
 
Tools for Development and Debugging in Python
Tools for Development and Debugging in PythonTools for Development and Debugging in Python
Tools for Development and Debugging in Python
zeeg
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deployment
zeeg
 
Building Scalable Web Apps
Building Scalable Web AppsBuilding Scalable Web Apps
Building Scalable Web Apps
zeeg
 
Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)
zeeg
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disqus
zeeg
 
Sentry (SF Python, Feb)
Sentry (SF Python, Feb)Sentry (SF Python, Feb)
Sentry (SF Python, Feb)
zeeg
 
Db tips & tricks django meetup
Db tips & tricks django meetupDb tips & tricks django meetup
Db tips & tricks django meetup
zeeg
 

More from zeeg (8)

Practicing Continuous Deployment
Practicing Continuous DeploymentPracticing Continuous Deployment
Practicing Continuous Deployment
 
Tools for Development and Debugging in Python
Tools for Development and Debugging in PythonTools for Development and Debugging in Python
Tools for Development and Debugging in Python
 
Pitfalls of Continuous Deployment
Pitfalls of Continuous DeploymentPitfalls of Continuous Deployment
Pitfalls of Continuous Deployment
 
Building Scalable Web Apps
Building Scalable Web AppsBuilding Scalable Web Apps
Building Scalable Web Apps
 
Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)Continuous Deployment at Disqus (Pylons Minicon)
Continuous Deployment at Disqus (Pylons Minicon)
 
PyCon 2011 Scaling Disqus
PyCon 2011 Scaling DisqusPyCon 2011 Scaling Disqus
PyCon 2011 Scaling Disqus
 
Sentry (SF Python, Feb)
Sentry (SF Python, Feb)Sentry (SF Python, Feb)
Sentry (SF Python, Feb)
 
Db tips & tricks django meetup
Db tips & tricks django meetupDb tips & tricks django meetup
Db tips & tricks django meetup
 

Recently uploaded

Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
Mariano Tinti
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
kumardaparthi1024
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
David Brossard
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
Recommendation System using RAG Architecture
Recommendation System using RAG ArchitectureRecommendation System using RAG Architecture
Recommendation System using RAG Architecture
fredae14
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Pixlogix Infotech
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
jpupo2018
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
tolgahangng
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 

Recently uploaded (20)

Mariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceXMariano G Tinti - Decoding SpaceX
Mariano G Tinti - Decoding SpaceX
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
GenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizationsGenAI Pilot Implementation in the organizations
GenAI Pilot Implementation in the organizations
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
Recommendation System using RAG Architecture
Recommendation System using RAG ArchitectureRecommendation System using RAG Architecture
Recommendation System using RAG Architecture
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERPBest 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Serial Arm Control in Real Time Presentation
Serial Arm Control in Real Time PresentationSerial Arm Control in Real Time Presentation
Serial Arm Control in Real Time Presentation
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 

DjangoCon 2010 Scaling Disqus

  • 1. Scaling the World’s Largest Django App Jason Yan David Cramer @jasonyan @zeeg
  • 3. What is DISQUS? dis·cuss • dĭ-skŭs' We are a comment system with an emphasis on connecting communities http://disqus.com/about/
  • 4. What is Scale? Number of Visitors 300M 250M 200M 150M 100M 50M Our traffic at a glance 17,000 requests/second peak 450,000 websites 15 million profiles 75 million comments 250 million visitors (August 2010)
  • 5. Our Challenges • We can’t predict when things will happen • Random celebrity gossip • Natural disasters • Discussions never expire • We can’t keep those millions of articles from 2008 in the cache • You don’t know in advance (generally) where the traffic will be • Especially with dynamic paging, realtime, sorting, personal prefs, etc.
  • 6. Our Challenges (cont’d) • High availability • Not a destination site • Difficult to schedule maintenance
  • 8. Server Architecture - Load Balancing • Load Balancing • High Availability • Software, HAProxy • heartbeat • High performance, intelligent server availability checking • Bonus: Nice statistics reporting Image Source: http://haproxy.1wt.eu/
  • 9. Server Architecture • ~100 Servers • 30% Web Servers (Apache + mod_wsgi) • 10% Databases (PostgreSQL) • 25% Cache Servers (memcached) • 20% Load Balancing / High Availability (HAProxy + heartbeat) • 15% Utility Servers (Python scripts)
  • 10. Server Architecture - Web Servers • Apache 2.2 • mod_wsgi • Using `maximum-requests` to plug memory leaks. • Performance Monitoring • Custom middleware (PerformanceLogMiddleware) • Ships performance statistics (DB queries, external calls, template rendering, etc) through syslog • Collected and graphed through Ganglia
  • 11. Server Architecture - Database • PostgreSQL • Slony-I for Replication • Trigger-based • Read slaves for extra read capacity • Failover master database for high availability
  • 12. Server Architecture - Database • Make sure indexes fit in memory and measure I/O • High I/O generally means slow queries due to missing indexes or indexes not in buffer cache • Log Slow Queries • syslog-ng + pgFouine + cron to automate slow query logging
  • 13. Server Architecture - Database • Use connection pooling • Django doesn’t do this for you • We use pgbouncer • Limits the maximum number of connections your database needs to handle • Save on costly opening and tearing down of new database connections
  • 15. Partitioning • Fairly easy to implement, quick wins • Done at the application level • Data is replayed by Slony • Two methods of data separation
  • 16. Vertical Partitioning Vertical partitioning involves creating tables with fewer columns and using additional tables to store the remaining columns. Forums Posts Users Sentry http://en.wikipedia.org/wiki/Partition_(database)
  • 17. Pythonic Joins Allows us to separate datasets posts = Post.objects.all()[0:25] # store users in a dictionary based on primary key users = dict( (u.pk, u) for u in User.objects.filter(pk__in=set(p.user_id for p in posts)) ) # map users to their posts for p in posts: p._user_cache = users.get(p.user_id)
  • 18. Pythonic Joins (cont’d) • Slower than at database level • But not enough that you should care • Trading performance for scale • Allows us to separate data • Easy vertical partitioning • More efficient caching • get_many, object-per-row cache
  • 19. Designating Masters • Alleviates some of the write load on your primary application master • Masters exist under specific conditions: • application use case • partitioned data • Database routers make this (fairly) easy
  • 20. Routing by Application class ApplicationRouter(object): def db_for_read(self, model, **hints): instance = hints.get('instance') if not instance: return None app_label = instance._meta.app_label return get_application_alias(app_label)
  • 21. Horizontal Partitioning Horizontal partitioning (also known as sharding) involves splitting one set of data into different tables. Disqus Your Blog CNN Telegraph http://en.wikipedia.org/wiki/Partition_(database)
  • 22. Horizontal Partitions • Some forums have very large datasets • Partners need high availability • Helps scale the write load on the master • We rely more on vertical partitions
  • 23. Routing by Partition class ForumPartitionRouter(object): def db_for_read(self, model, **hints): instance = hints.get('instance') if not instance: return None forum_id = getattr(instance, 'forum_id', None) if not forum_id: return None return get_forum_alias(forum_id) # What we used to do Post.objects.filter(forum=forum) # Now, making sure hints are available forum.post_set.all()
  • 24. Optimizing QuerySets • We really dislike raw SQL • It creates more work when dealing with partitions • Built-in cache allows sub-slicing • But isn’t always needed • We removed this cache
  • 25. Removing the Cache • Django internally caches the results of your QuerySet • This adds additional memory overhead # 1 query qs = Model.objects.all()[0:100] # 0 queries (we don’t need this behavior) qs = qs[0:10] # 1 query qs = qs.filter(foo=bar) • Many times you only need to view a result set once • So we built SkinnyQuerySet
  • 26. Removing the Cache (cont’d) Optimizing memory usage by removing the cache class SkinnyQuerySet(QuerySet): def __iter__(self): if self._result_cache is not None: # __len__ must have been run return iter(self._result_cache) has_run = getattr(self, 'has_run', False) if has_run: raise QuerySetDoubleIteration("...") self.has_run = True # We wanted .iterator() as the default return self.iterator() http://gist.github.com/550438
  • 27. Atomic Updates • Keeps your data consistent • save() isnt thread-safe • use update() instead • Great for things like counters • But should be considered for all write operations
  • 28. Atomic Updates (cont’d) Thread safety is impossible with .save() Request 1 post = Post(pk=1) # a moderator approves post.approved = True post.save() Request 2 post = Post(pk=1) # the author adjusts their message post.message = ‘Hello!’ post.save()
  • 29. Atomic Updates (cont’d) So we need atomic updates Request 1 post = Post(pk=1) # a moderator approves Post.objects.filter(pk=post.pk) .update(approved=True) Request 2 post = Post(pk=1) # the author adjusts their message Post.objects.filter(pk=post.pk) .update(message=‘Hello!’)
  • 30. Atomic Updates (cont’d) A better way to approach updates def update(obj, using=None, **kwargs): """ Updates specified attributes on the current instance. """ assert obj, "Instance has not yet been created." obj.__class__._base_manager.using(using) .filter(pk=obj) .update(**kwargs) for k, v in kwargs.iteritems(): if isinstance(v, ExpressionNode): # NotImplemented continue setattr(obj, k, v) http://github.com/andymccurdy/django-tips-and-tricks/blob/master/model_update.py
  • 31. Delayed Signals • Queueing low priority tasks • even if they’re fast • Asynchronous (Delayed) signals • very friendly to the developer • ..but not as friendly as real signals
  • 32. Delayed Signals (cont’d) We send a specific serialized version of the model for delayed signals from disqus.common.signals import delayed_save def my_func(data, sender, created, **kwargs): print data[‘id’] delayed_save.connect(my_func, sender=Post) This is all handled through our Queue
  • 33. Caching • Memcached • Use pylibmc (newer libMemcached-based) • Ticket #11675 (add pylibmc support) • Third party applications: • django-newcache, django-pylibmc
  • 34. Caching (cont’d) • libMemcached / pylibmc is configurable with “behaviors”. • Memcached “single point of failure” • Distributed system, but we must take precautions. • Connection timeout to memcached can stall requests. • Use `_auto_eject_hosts` and `_retry_timeout` behaviors to prevent reconnecting to dead caches.
  • 35. Caching (cont’d) • Default (naive) hashing behavior • Modulo hashed cache key cache for index to server list. • Removal of a server causes majority of cache keys to be remapped to new servers. CACHE_SERVERS = [‘10.0.0.1’, ‘10.0.0.2’] key = ‘my_cache_key’ cache_server = CACHE_SERVERS[hash(key) % len(CACHE_SERVERS)]
  • 36. Caching (cont’d) • Better approach: consistent hashing • libMemcached (pylibmc) uses libketama (http://tinyurl.com/lastfm-libketama) • Addition / removal of a cache server remaps (K/n) cache keys (where K=number of keys and n=number of servers) Image Source: http://sourceforge.net/apps/mediawiki/kai/index.php?title=Introduction
  • 37. Caching (cont’d) • Thundering herd (stampede) problem • Invalidating a heavily accessed cache key causes many clients to refill cache. • But everyone refetching to fill the cache from the data store or reprocessing data can cause things to get even slower. • Most times, it’s ideal to return the previously invalidated cache value and let a single client refill the cache. • django-newcache or MintCache (http:// djangosnippets.org/snippets/793/) will do this for you. • Prefer filling cache on invalidation instead of deleting from cache also helps to prevent the thundering herd problem.
  • 38. Transactions • TransactionMiddleware got us started, but down the road became a burden • For postgresql_psycopg2, there’s a database option, OPTIONS[‘autocommit’] • Each query is in its own transaction. This means each request won’t start in a transaction. • But sometimes we want transactions (e.g., saving multiple objects and rolling back on error)
  • 39. Transactions (cont’d) • Tips: • Use autocommit for read slave databases. • Isolate slow functions (e.g., external calls, template rendering) from transactions. • Selective autocommit • Most read-only views don’t need to be in transactions. • Start in autocommit and switch to a transaction on write.
  • 40. Scaling the Team • Small team of engineers • Monthly users / developers = 40m • Which means writing tests.. • ..and having a dead simple workflow
  • 41. Keeping it Simple • A developer can be up and running in a few minutes • assuming postgres and other server applications are already installed • pip, virtualenv • settings.py
  • 42. Setting Up Local 1. createdb -E UTF-8 disqus 2. git clone git://repo 3. mkvirtualenv disqus 4. pip install -U -r requirements.txt 5. ./manage.py syncdb && ./manage.py migrate
  • 43. Sane Defaults settings.py from disqus.conf.settings.default import * try: from local_settings import * except ImportError: import sys, traceback sys.stderr.write("Can't find 'localsettings.py’n”) sys.stderr.write("nThe exception was:nn") traceback.print_exc() local_settings.py from disqus.conf.settings.dev import *
  • 44. Continuous Integration • Daily deploys with Fabric • several times an hour on some days • Hudson keeps our builds going • combined with Selenium • Post-commit hooks for quick testing • like Pyflakes • Reverting to a previous version is a matter of seconds
  • 45. Continuous Integration (cont’d) Hudson makes integration easy
  • 46. Testing • It’s not fun breaking things when you’re the new guy • Our testing process is fairly heavy • 70k (Python) LOC, 73% coverage, 20 min suite • Custom Test Runner (unittest) • We needed XML, Selenium, Query Counts • Database proxies (for read-slave testing) • Integration with our Queue
  • 47. Testing (cont’d) Query Counts # failures yield a dump of queries def test_read_slave(self): Model.objects.using(‘read_slave’).count() self.assertQueryCount(1, ‘read_slave’) Selenium def test_button(self): self.selenium.click('//a[@class=”dsq-button”]') Queue Integration class WorkerTest(DisqusTest): workers = [‘fire_signal’] def test_delayed_signal(self): ...
  • 48. Bug Tracking • Switched from Trac to Redmine • We wanted Subtasks • Emailing exceptions is a bad idea • Even if its localhost • Previously using django-db-log to aggregate errors to a single point • We’ve overhauled db log and are releasing Sentry
  • 49. django-sentry Groups messages intelligently http://github.com/dcramer/django-sentry
  • 50. django-sentry (cont’d) Similar feel to Django’s debugger http://github.com/dcramer/django-sentry
  • 51. Feature Switches • We needed a safety in case a feature wasn’t performing well at peak • it had to respond without delay, globally, and without writing to disk • Allows us to work out of trunk (mostly) • Easy to release new features to a portion of your audience • Also nice for “Labs” type projects
  • 53. Final Thoughts • The language (usually) isn’t your problem • We like Django • But we maintain local patches • Some tickets don’t have enough of a following • Patches, like #17, completely change Django.. • ..arguably in a good way • Others don’t have champions Ticket #17 describes making the ORM an identify mapper
  • 54. Housekeeping Birds of a Feather Want to learn from others about performance and scaling problems? Or play some StarCraft 2? We’re Hiring! DISQUS is looking for amazing engineers
  • 56. References django-sentry http://github.com/dcramer/django-sentry Our Feature Switches http://cl.ly/2FYt Andy McCurdy’s update() http://github.com/andymccurdy/django-tips-and-tricks Our PyFlakes Fork http://github.com/dcramer/pyflakes SkinnyQuerySet http://gist.github.com/550438 django-newcache http://github.com/ericflo/django-newcache attach_foreignkey (Pythonic Joins) http://gist.github.com/567356

Editor's Notes

  1. Hi. I'm Jason (and I'm David), and we're from Disqus.
  2. Show of hands, How many of you know what DISQUS is?
  3. For those of you who are not familiar with us, DISQUS is a comment system that focuses on connecting communities. We power discussions on such sites as CNN, IGN, and more recently Engadget and TechCrunch. Our company was founded back in 2007 by my co-founder, Daniel Ha, and I back where we started working out of our dorm room. Our decision to use Django came down primarily to our dislike for PHP which we were previously using. Since then, we've grown Disqus to over 250+ million visitors a month.
  4. We've peaked at over 17,000 requests per second, to Django, and we currently power comments on nearly half a million websites which accounts for more than 15 million profiles who have left over 75 million comments.
  5. As you can imagine we have some big challenges when it comes to scaling a large Django application. For one, it’s hard to predict when events happen like last year with Michael Jackson’s death, and more recently, the Gulf Oil Spill. Another challenge we have is the fact that discussions never expire. When you visit that blog post from 2008 we have to be ready to serve those comments immediately. Not only does THAT make caching difficult, but we also have to deal with things such as dynamic paging, realtime commenting, and other personal preferences. This makes it even more important to be able to serve those quickly without relying on the cache.
  6. So we also have some interesting infrastructure problems when it comes to scaling Disqus. We're not a destination website, so if we go down, it affects other sites as well as ours. Because of this, it's difficult for us to schedule maintenance, so we face some interesting scaling and availbility challenges.
  7. As you can see, we have tried to keep the stack pretty thin. This is because, as we've learned, the more services we try to add, the more difficult it is to support. And especially because we have a small team, this becomes difficult to manage. So we use DNS load balancing to spread the requests to multiple HAProxy servers which are our software load balancers. These proxy requests to our backend app servers which run mod_wsgi. We use memcache for caching, and we have a custom wrapper using syslog for our queue. For our data store, we use PostgreSQL, and for replication, we use Slony for failover and read slaves.
  8. As I said, we use HAProxy for HTTP load balancing. It's a high performance software load balancer with intelligent failure detection. It also provides you with nice statistics of your requests. We use heartbeat for high availability and we have it take over the IP address of the down machine.
  9. We have about 100GB of cache. Because of our high availability requirements, 20% are allocated to high availability and load balancing.
  10. Our web servers are pretty standard. We use mod_wsgi mostly because it just works. Performance wise, you're really going to be bottlenecked on the application. The cool thing we do is that we actually hasve a custom middleware that does performance monitoring. What this does is ship data from our application about external calls like database, cache calls, and we collect it and graph it with Ganglia.
  11. The more interesting aspect of our server architecture is how we have our database setup. As I mentioned, we use Postgres as our database. Honestly, we used it because Django recommended it, and my recommendation is that if you’re not already an expert in a database, you're better off going with Postgres. We use slony for replication Slony is trigger-based which means that every write is captured and strored in a log table and those events are replayed to slave databases. This is nice over otehr methods such as log shipping because it allows us to have flexible schemas across read lsaves. For example, some of our read slaves have different indexes. We also use slony for failover for high availbility.
  12. There are a few things we do to keep our database healthy. We keep our indexes in memory, and when we can't, we partition our data. We also have application-specific indexes on our readslaves. Another important thing we've done is measure I/O. Any time we've seen high I/O is usually because we're missing indexes or indexes aren't fitting in memory. Lastly, we monitor slow queries. We send logs to pgfouine via syslog which genererates a nice report showing you which queries are the slowest.
  13. The last thing we've found to be really helpful is switching to database connection pool. Remember, Django doesn't do this for you. We use pgbouncer for this, and there are a few easy wins for using it. One is that it limits the maximum connections to the database so it doesn't have to handle as many concurrent connections. Secpondly, you save the cost of opening and tearing down new connections per request.
  14. Moving on to our application, we’ve found that most of the struggle is with the database layer. We’ve got a pretty standard layout if you’re familiar with forums. Forum has many threads, which has many posts. Posts use an adjacency list model, and also reference Users. With this kind of data model, one of our quickest wins has been the ability to partition data.
  15. It’s almost entirely done at the application level, which makes it fairly easy to implement. The only thing not handled by the app is replication, and Slony does that for us. We handle partitioning in a couple of ways.
  16. The first of which are vertical partitions. This is probably the simplest thing you can implement in your application. Kill off your joins and spread out your applications on multiple databases. Some database engines might make this easier than others, but Slony allows us to easily replicate very specific data.
  17. Using this method you’ll need to handle joins in your Python application. We do this by performing two separate queries and mapping the foreign keys to the parent objects. For us the easiest way has been to throw them into a dictionary, iterate through the other queryset, and set the foreignkey cache’s value to the instance.
  18. A few things to keep in mind when doing pythonic joins. They’re not going to be as fast in the database. You can’t avoid this, but it’s not something you should worry about. With this however, you get plain and simple vertical partitions. You also can cache things a lot easier, and more efficiently fetch them using things like get_many and a singular object cache. Overall your’e trading performance for scale.
  19. Another benefit that comes from vertical partitioning is the ability to designate masters. We do this to alleviate some of the load on our primary application master. So for example, server FOO might be the source for writes on the Users table, while server BAR handles all of our other forum data. Since we’re using Django 1.2 we also get routing for free through the new routers.
  20. Here’s an example of a simple application router. It let’s us specify a read-slave based on our app label. So if its users, we go to FOO, if its forums, we go to BAR. You can handle this logic any way you want, pretty simple and powerful.
  21. While we use vertical partitioning for most cases, eventually you hit an issue where your data just doesn’t scale on a single database. You’re probably familiar with the word sharding, well that’s what we do with our forum data. We’ve set it up so that we can send certain large sites to dedicated machines. This also uses designated masters as we mentioned with the other partitions.
  22. We needed this when write and read load combined became so big that it was just hard to keep up on a single set of machines. It also gives the nice added benefit of high availability in many situations. Mostly though, it all goes back to scaling our master databases.
  23. So again we’re using the router here to handle partitioning of the forums. We can specify that CNN goes to this database alias, which could be any number of machines, and everything else goes to our default cluster. The one caveat we found with this, is sometimes hints aren’t present in the router. I believe within the current version of Django they are only available when using a relational lookup, such as a foreign key. All in all it’s pretty powerful, and you just need to be aware of it while writing your queries.