Web Framework Performance - Examples from Django and Rails

8,629 views

Published on

Video and slides synchronized, mp3 and slide download available at http://bit.ly/136ggDI.

Gareth Rushgrove overviews Ruby on Rails and Django: object caches, fragment and HTTP caching, asset compilation, profiling, log file measurement and framework hooks for instrumentation. Filmed at qconsf.com.

Gareth Rushgrove is now a developer at Government Digital Service, part of the UK Government. Gareth has written articles on topics from mobile web design to Facebook and website performance to JavaScript for the likes of Vitamin, Digital Web and Opera. He also featured in the 2007 edition of 24ways, the annual web design advent calendar. Blog: www.garethrushgrove.com Twittter: @garethr

Published in: Technology

Web Framework Performance - Examples from Django and Rails

  1. 1. Web Framework PerformanceExamples from Django and RailsQConSF 9th November 2012gareth rushgrove | morethanseven.net www.flickr.com/photos/mugley/5013931959/
  2. 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /Django-Rails InfoQ.com: News & Community Site• 750,000 unique visitors/month• Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese)• Post content from our QCon conferences• News 15-20 / week• Articles 3-4 / week• Presentations (videos) 12-15 / week• Interviews 2-3 / week• Books 1 / month
  3. 3. Presented at QCon San Francisco www.qconsf.comPurpose of QCon- to empower software development by facilitating the spread ofknowledge and innovationStrategy - practitioner-driven conference designed for YOU: influencers ofchange and innovation in your teams- speakers and topics driving the evolution and innovation- connecting and catalyzing the influencers and innovatorsHighlights- attended by more than 12,000 delegates since 2007- held in 9 cities worldwide
  4. 4. Me
  5. 5. Gareth Rushgrovegareth rushgrove | morethanseven.net
  6. 6. Curate devopsweekly.comgareth rushgrove | morethanseven.net
  7. 7. Blog at morethanseven.netgareth rushgrove | morethanseven.net
  8. 8. TextWork at UK Government Digital Servicegareth rushgrove | morethanseven.net
  9. 9. I am a Civil Servantgareth rushgrove | morethanseven.net http://www.flickr.com/photos/benterrett/6852348725/
  10. 10. Developer, Operations, Product Guygareth rushgrove | morethanseven.net
  11. 11. Introduction(what’s the problem) http://www.flickr.com/photos/iancarroll/5027441664
  12. 12. Slow(er) languages...gareth rushgrove | morethanseven.net
  13. 13. Slow(er) languages...gareth rushgrove | morethanseven.net
  14. 14. ...don’t mean slow applicationsgareth rushgrove | morethanseven.net
  15. 15. ...don’t have to mean slow applicationsgareth rushgrove | morethanseven.net
  16. 16. Frameworks can helpgareth rushgrove | morethanseven.net
  17. 17. Frameworks can helpgareth rushgrove | morethanseven.net
  18. 18. A real examplegareth rushgrove | morethanseven.net
  19. 19. A sample applicationgareth rushgrove | morethanseven.net
  20. 20. 49msBeforegareth rushgrove | morethanseven.net
  21. 21. 6msAftergareth rushgrove | morethanseven.net
  22. 22. - Analyze your application - Know your framework - Cache everywhere - Instrument everything - Don’t just think about developmentThis presentationgareth rushgrove | morethanseven.net
  23. 23. Analyze(count everything) http://www.flickr.com/photos/iancarroll/5027441664
  24. 24. 1. From the browsergareth rushgrove | morethanseven.net
  25. 25. YSlowgareth rushgrove | morethanseven.net
  26. 26. webpagetest.orggareth rushgrove | morethanseven.net
  27. 27. 2. From the codegareth rushgrove | morethanseven.net
  28. 28. Django debug toolbargareth rushgrove | morethanseven.net
  29. 29. Django debug toolbargareth rushgrove | morethanseven.net
  30. 30. Django debug toolbargareth rushgrove | morethanseven.net
  31. 31. Django debug toolbargareth rushgrove | morethanseven.net
  32. 32. Profiling middlewaregareth rushgrove | morethanseven.net
  33. 33. rack-mini-profilergareth rushgrove | morethanseven.net
  34. 34. rack-mini-profiler detailsgareth rushgrove | morethanseven.net
  35. 35. Rack Insightgareth rushgrove | morethanseven.net
  36. 36. Rack Insightgareth rushgrove | morethanseven.net
  37. 37. Rails footnotesgareth rushgrove | morethanseven.net
  38. 38. New Relic development modegareth rushgrove | morethanseven.net
  39. 39. 3. Logsgareth rushgrove | morethanseven.net
  40. 40. Request log analyzergareth rushgrove | morethanseven.net
  41. 41. ┃ Mean ┃ StdDev ┃ Min ┃ Max ┃ 95 %tile ┃ ┃ 0.16s ┃ 0.26s ┃ 0.01s ┃ 1.74s ┃ 0.01s-1.08s ┃Request log analyzer detailsgareth rushgrove | morethanseven.net
  42. 42. Django timeloggareth rushgrove | morethanseven.net
  43. 43. Know your framework(use the parts you need) http://www.flickr.com/photos/iancarroll/5027441664
  44. 44. 1. Disable what you don’t needgareth rushgrove | morethanseven.net
  45. 45. require "rails/all" in config/application.rbDon’t import railsgareth rushgrove | morethanseven.net
  46. 46. require "action_controller/railtie" require "rails/test_unit/railtie" require "sprockets/railtie" in config/application.rbJust the bits you needgareth rushgrove | morethanseven.net
  47. 47. MIDDLEWARE_CLASSES = ( django.middleware.common.CommonMiddleware, django.contrib.sessions.middleware.SessionMiddleware, django.middleware.csrf.CsrfViewMiddleware, django.contrib.auth.middleware.AuthenticationMiddleware, django.contrib.messages.middleware.MessageMiddleware, ) in settings.pyDjango middlewaregareth rushgrove | morethanseven.net
  48. 48. INSTALLED_APPS = ( django.contrib.auth, django.contrib.contenttypes, django.contrib.sessions, django.contrib.sites, django.contrib.messages, django.contrib.staticfiles, ) in settings.pyDjango installed appsgareth rushgrove | morethanseven.net
  49. 49. 2. Know your ORMgareth rushgrove | morethanseven.net
  50. 50. Django select_related()gareth rushgrove | morethanseven.net
  51. 51. queryset = Release.objects.all() 11 SQL QueriesN+1 problemgareth rushgrove | morethanseven.net
  52. 52. queryset = Release.objects.select_related() 1 SQL QueryJoins to the rescuegareth rushgrove | morethanseven.net
  53. 53. Active Record Includegareth rushgrove | morethanseven.net
  54. 54. Release.all 11 SQL QueriesN+1 problemgareth rushgrove | morethanseven.net
  55. 55. Release.includes(:app).all 1 SQL QueryJoins to the rescuegareth rushgrove | morethanseven.net
  56. 56. <link href="/assets/application.css?body=1" ... <link href="/assets/apps.css?body=1" ... <link href="/assets/bootstrap_and_overrides.css?body=1" ... <link href="/assets/releases.css?body=1" ... <link href="/assets/sample.css?body=1" ... <link href="/assets/scaffolds.css?body=1" ... to this <link href="/assets/application.css" ...3. Asset compilation in HTMLgareth rushgrove | morethanseven.net
  57. 57. h1 { padding-top: 40px; } /*! * Bootstrap v2.2.1 * * Copyright 2012 Twitter, Inc * Licensed under the Apache License v2.0 * http://www.apache.org/licenses/LICENSE-2.0 * * Designed and built with all the love in the world @twitter by @mdo and @fat. */ article, aside, details, figcaption, figure, footer, header, hgroup, nav, section { display: block; } to this h1{padding-top:40px} article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{displ ay:block}Asset compilation in CSSgareth rushgrove | morethanseven.net
  58. 58. # Enable the asset pipeline config.assets.enabled = true # Version of your assets config.assets.version = 1.0 in config/application.rb # compress assets config.assets.compress = true # Don’t expands the lines which load the assets config.assets.debug = false in config/environments/production.rbAsset compilation configurationgareth rushgrove | morethanseven.net
  59. 59. 4. Different runtimesgareth rushgrove | morethanseven.net
  60. 60. Different runtimes - JRubygareth rushgrove | morethanseven.net
  61. 61. Different runtimes - PyPygareth rushgrove | morethanseven.net
  62. 62. require test_helper require rails/performance_test_help class ReleaseTest < ActionDispatch::PerformanceTest self.profile_options = { :runs => 10, :metrics => [:wall_time]} def test_release_index get /releases end end in test/performance/releases_test.rb5. Performance testsgareth rushgrove | morethanseven.net
  63. 63. rake test:benchmark Started ReleaseTest#test_release_index (110 ms warmup) wall_time: 6 ms Finished in 0.551815 seconds. 1 tests, 0 assertions, 0 failures, 0 errors, 0 skipsRails benchmarkergareth rushgrove | morethanseven.net
  64. 64. Cache(everything) http://www.flickr.com/photos/iancarroll/5027441664
  65. 65. 1. Built-in caching supportgareth rushgrove | morethanseven.net
  66. 66. Great documentation 1gareth rushgrove | morethanseven.net
  67. 67. Great documentation 2gareth rushgrove | morethanseven.net
  68. 68. 2. ORM Cachinggareth rushgrove | morethanseven.net
  69. 69. CACHES = { default : dict( BACKEND = johnny.backends.memcached.MemcachedCache LOCATION = [127.0.0.1:11211], JOHNNY_CACHE = True, ) } MIDDLEWARE_CLASSES = ( johnny.middleware.LocalStoreClearMiddleware, johnny.middleware.QueryCacheMiddleware, ) + MIDDLEWARE_CLASSES in test/performance/releases_test.rbJohnny Cachegareth rushgrove | morethanseven.net
  70. 70. First request 4 Queriesgareth rushgrove | morethanseven.net
  71. 71. Subsequent requests 0 Queriesgareth rushgrove | morethanseven.net
  72. 72. class Release < ActiveRecord::Base acts_as_cached after_save :expire_cache attr_accessible :app_id belongs_to :app def self.recent includes(:app).al end endCache-fugareth rushgrove | morethanseven.net
  73. 73. class Release < ActiveRecord::Base acts_as_cached after_save :expire_cache attr_accessible :app_id belongs_to :app def self.recent includes(:app).al end endAdd cache behaviourgareth rushgrove | morethanseven.net
  74. 74. @releases = Release.cached(:recent)Get cached contentgareth rushgrove | morethanseven.net
  75. 75. First request 2 Queriesgareth rushgrove | morethanseven.net
  76. 76. Subsequent requests 0 Queriesgareth rushgrove | morethanseven.net
  77. 77. http://www.mnot.net/cache_docs/3. HTTP is your friendgareth rushgrove | morethanseven.net
  78. 78. from django.views.decorators.cache import cache_control @cache_control(public=True, max_age=3600)HTTP headersgareth rushgrove | morethanseven.net
  79. 79. class ReleasesController < ApplicationController def index expires_in 60.minute, :public => true in app/controllers/releases_controller.rbHTTP headersgareth rushgrove | morethanseven.net
  80. 80. INSTALLED_APPS = INSTALLED_APPS + ( varnishapp, ) VARNISH_MANAGEMENT_ADDRS = ( localhost:6082, ) VARNISH_WATCHED_MODELS = (app.release,) in settings.pydjango-varnish configurationgareth rushgrove | morethanseven.net
  81. 81. django-varnish admingareth rushgrove | morethanseven.net
  82. 82. Instrument(monitor all the things) http://www.flickr.com/photos/iancarroll/5027441664
  83. 83. MIDDLEWARE_CLASSES = MIDDLEWARE_CLASSES + ( django_statsd.middleware.GraphiteRequestTimingMiddleware, django_statsd.middleware.GraphiteMiddleware, ) STATSD_PATCHES = [ django_statsd.patches.db, django_statsd.patches.cache, ] INSTALLED_APPS = INSTALLED_APPS + ( django_statsd, ) from settings.pydjango-statsdgareth rushgrove | morethanseven.net
  84. 84. django-statsd on debug toolbargareth rushgrove | morethanseven.net
  85. 85. MIDDLEWARE_CLASSES = ( django_mmstats.middleware.MmStatsMiddleware, ) import stats MMSTATS_CLASS = stats.DjangoStats from settings.py from django_mmstats.base import BaseDjangoStats class DjangoStats(BaseDjangoStats): """Add mmstats fields here, just like Django models!""" from stats.pydjango-mmstatsgareth rushgrove | morethanseven.net
  86. 86. mmashgareth rushgrove | morethanseven.net
  87. 87. librato-railsgareth rushgrove | morethanseven.net
  88. 88. Not just indevelopment(measure in production) http://www.flickr.com/photos/iancarroll/5027441664
  89. 89. 1. Logs (again)gareth rushgrove | morethanseven.net
  90. 90. Logstergareth rushgrove | morethanseven.net
  91. 91. Logragegareth rushgrove | morethanseven.net
  92. 92. Started GET "/" for 127.0.0.1 at 2012-03-10 14:28:14 +0100 Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms | ActiveRecord: 0.0ms) to this GET /jobs/833552.json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26Logragegareth rushgrove | morethanseven.net
  93. 93. Logstashgareth rushgrove | morethanseven.net
  94. 94. LOGRAGE %{WORD:method}%{SPACE}%{DATA}% {SPACE}action=%{WORD:controller}#% {WORD:action}%{SPACE}status=%{INT:status}% {SPACE}duration=%{NUMBER:duration}%{SPACE} view=%{NUMBER:view}(%{SPACE}db=% {NUMBER:db})?%{GREEDYDATA}Logstash plus logragegareth rushgrove | morethanseven.net
  95. 95. LOGRAGE %{WORD:method}%{SPACE}%{DATA}% {SPACE}action=%{WORD:controller}#% {WORD:action}%{SPACE}status=%{INT:status}% {SPACE}duration=%{NUMBER:duration}%{SPACE} view=%{NUMBER:view}(%{SPACE}db=% {NUMBER:db})?%{GREEDYDATA}Rails performance datagareth rushgrove | morethanseven.net
  96. 96. output { statsd { host => "localhost" tags => [ "lograge" ] timing => [ "<%= @title %>.%{controller}.%{action}.% {method}.duration", "%{duration}" ] } statsd { host => “localhost” tags => [ "lograge" ] timing => [ "<%= @title %>.%{controller}.%{action}.% {method}.view", "%{view}" ] } }Output to statsdgareth rushgrove | morethanseven.net
  97. 97. 2. Metricsgareth rushgrove | morethanseven.net
  98. 98. Gangliagareth rushgrove | morethanseven.net
  99. 99. Graphitegareth rushgrove | morethanseven.net
  100. 100. Riemanngareth rushgrove | morethanseven.net
  101. 101. New Relic newrelic.comgareth rushgrove | morethanseven.net
  102. 102. Librato Metrics metrics.librato.comgareth rushgrove | morethanseven.net
  103. 103. Conclusions(if all you remember is) http://www.flickr.com/photos/iancarroll/5027441664
  104. 104. Tooling helps. A lot.gareth rushgrove | morethanseven.net
  105. 105. - A debug toolbar - Transparent caching support - Hooks for instrumentation - Configurable loggingYour framework should havegareth rushgrove | morethanseven.net
  106. 106. If not... build themgareth rushgrove | morethanseven.net
  107. 107. Share everythinggareth rushgrove | morethanseven.net
  108. 108. The End
  109. 109. www.flickr.com/photos/snugglepup/Thanks for the amazing photosgareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/
  110. 110. Questions?gareth rushgrove | morethanseven.net http://flickr.com/photos/psd/102332391/

×