Lessons Learnt.  An In Production,Scaleable Django Stack PyCon ireland - Dublin, Oct 13th-14th 2012         john.moylan@rt...
Brief Intro to RTE.ieWebsite of Irelands national broadcaster
ProblemMostly W.O.R.M, but some stuff cant be cached :(Lots of content(100s GBs of HTML)10s TB of Audio and VideoLots of s...
Traffic GrowthOct 2010 Vs Oct 2011Traffic nearly doubled.
Flashmob Traffic36Gb/s of Traffic. 41K Concurrent streams.Playlists were not cacheable........But what a great fight.
Static?Legacy system published to static files usingXML/XSL Publisher.Multiple versions per type of view.SSIs inside SSIs ...
Dynamic RequirementEvaluated some systems:Wicket, Rails, Shale....DjangoHmmm... niceAll in package, sensible defaults. Sim...
Try Before you buy2008Wrote prototype video upload sharing site...(Django 0.96) and then decided to make it live.http://we...
Try Before You Buy2009Original RTÉ Player - Written in Django.2011Lets use Django to make the whole sitedynamic!
..continued2012Sport, About, RTÉ Player v.2, Radio, News?,Archives, playlists, lifestyle, SSO......and onand on2013..More
Django DeliveryLinuxApache with Event MPM and mod_wsgiMysql and MHA (tip Innodb plugin)Solr for search and NoSQLVarnish (u...
Code ManagementGitRedmineGitosis
DeploymentCobblerFabricBuildout
MonitoringMonitor & Measure everything!ZenossRRDToolMRTGGraphiteStatsd - mozillas Django-statsdSentryLots of custom monito...
In House Monitoring projectsPyvarnishhttps://github.com/redsnapper8t8/pyvarnish
In House Monitoring ProjectsDigital Dash - (open sourced soon)
BenchmarkingTool agnostic but seem to use these alot.Bonnie++Netperf#HTTPHttperfSiege
Performance Tips1.   Memory, memory, memory2.   Caching - in memory3.   Does it scale horizontally?4.   KISS
Lessons Learnt.Read from memory whenever possibleCache, learned this one a long time ago butwhenever we have issues it can...
Thank youQuestions?Also, did I mention that we are hiring?
Upcoming SlideShare
Loading in …5
×

Pycon2012

443 views

Published on

My PyCon Ireland 2012 talk with an analysis and description of some of the various in production parts of our stack.

IE, Linux, MySQL, Memcache, Django, Apache, Solr, Django-sentry, Buildout/Fabric, statsd etc.

Some discussion also of alternative choices, up and coming technology, things to watch.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
443
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Pycon2012

  1. 1. Lessons Learnt. An In Production,Scaleable Django Stack PyCon ireland - Dublin, Oct 13th-14th 2012 john.moylan@rte.ie / john@8t8.eu / @redsnapper88
  2. 2. Brief Intro to RTE.ieWebsite of Irelands national broadcaster
  3. 3. ProblemMostly W.O.R.M, but some stuff cant be cached :(Lots of content(100s GBs of HTML)10s TB of Audio and VideoLots of strands (News, Sport, Business, Ent., TV, Radio,Teletext, Archives, Young Peoples, Performing Groups...)Small team.Try to keep things as simple and efficient as possible. ...massive growth
  4. 4. Traffic GrowthOct 2010 Vs Oct 2011Traffic nearly doubled.
  5. 5. Flashmob Traffic36Gb/s of Traffic. 41K Concurrent streams.Playlists were not cacheable........But what a great fight.
  6. 6. Static?Legacy system published to static files usingXML/XSL Publisher.Multiple versions per type of view.SSIs inside SSIs inside SSIsRepublishing a strand could take days.
  7. 7. Dynamic RequirementEvaluated some systems:Wicket, Rails, Shale....DjangoHmmm... niceAll in package, sensible defaults. Similarproblem domain.
  8. 8. Try Before you buy2008Wrote prototype video upload sharing site...(Django 0.96) and then decided to make it live.http://web.archive.org/web/20090318064215/http://www.rte.ie/tv/theden/ugc/red/video/
  9. 9. Try Before You Buy2009Original RTÉ Player - Written in Django.2011Lets use Django to make the whole sitedynamic!
  10. 10. ..continued2012Sport, About, RTÉ Player v.2, Radio, News?,Archives, playlists, lifestyle, SSO......and onand on2013..More
  11. 11. Django DeliveryLinuxApache with Event MPM and mod_wsgiMysql and MHA (tip Innodb plugin)Solr for search and NoSQLVarnish (used to use Squid)LVS-NAT and LVS-DR Load-balancers.MemcacheDeploying Glusterfs
  12. 12. Code ManagementGitRedmineGitosis
  13. 13. DeploymentCobblerFabricBuildout
  14. 14. MonitoringMonitor & Measure everything!ZenossRRDToolMRTGGraphiteStatsd - mozillas Django-statsdSentryLots of custom monitoring
  15. 15. In House Monitoring projectsPyvarnishhttps://github.com/redsnapper8t8/pyvarnish
  16. 16. In House Monitoring ProjectsDigital Dash - (open sourced soon)
  17. 17. BenchmarkingTool agnostic but seem to use these alot.Bonnie++Netperf#HTTPHttperfSiege
  18. 18. Performance Tips1. Memory, memory, memory2. Caching - in memory3. Does it scale horizontally?4. KISS
  19. 19. Lessons Learnt.Read from memory whenever possibleCache, learned this one a long time ago butwhenever we have issues it can usually be trackeddown bad design and lack of caching. Dont hityour web servers unless you have to.Memcache, VarnishLong live Rest - GETs should not have sideeffects. VarnishKISS - and no resume driven development.
  20. 20. Thank youQuestions?Also, did I mention that we are hiring?

×