SCALABLE WEB APPLICATION
ARCHITECTURE
(USING DJANGO + EXTJS)
Michał Karzyński, PyWaw #27
TALK OUTLINE
1. Application architecture
2. Backend code organization (Django)
3. Frontend code organization (ExtJS)
4. Co...
YOURSTRULY
Michał Karzyński
• project manager at Politechnika Gdańska
• freelance developer and consultant
• polyglot, cur...
APPLICATION ARCHITECTURE
Backend API Frontend
Server Cloud Browser
Python JSON JavaScript
Django ExtJS
I/O GUITransport
BACKEND (DJANGO)
CODE STRUCTURE
Decorators
Router
Middleware
Controller (a.k.aView)
Template / Serializer
Middleware
HTTP ...
MIDDLEWARE
Decorators
Router
Middleware
Controller (a.k.aView)
Template / Serializer
Middleware
Model (ORM)
• Processes in...
MIDDLEWARE
EXAMPLE
class SessionMiddleware(object):
def process_request(self, request):
engine = import_module(settings.SE...
ROUTER
• Maps URL patterns to
controller function calls
• Uses regular expressions
• Can use positional and named
paramete...
ROUTER
EXAMPLE
urlpatterns = patterns('myapp.views',
url(r'^$', home_page), # simple pattern
url(r'^admin/', include(admin...
DECORATORS
• Wrap controller functions in
additional logic
• Can modify request object or
cause a redirect
• e.g. login_re...
CONTROLLER (VIEW)
• Main application logic
• Functions receive an HTTP
request and returns an HTTP
response
• Communicates...
CONTROLLER (VIEW)
EXAMPLE
def display_institute(request, id):
institute = get_object_or_404(Institute, pk=safeint(id))
ret...
CONTROLLER (VIEW)
EXAMPLE
def save_offer(self, request):
form_data = json_deserialize(request.read())
# Validate incoming ...
MODEL (ORM)
• Database abstraction layer
• Converts DB rows into
Python objects
• Generates DB structure
and lazy-loading ...
MODEL (ORM)
EXAMPLE
class Person(models.Model):
first_name = models.CharField(_('first name'), max_length=30, blank=True)
...
TEMPLATE
• Generate HTML
• Use Django’s templating
language
Decorators
Router
Middleware
Controller (a.k.aView)
Template /...
TEMPLATE
EXAMPLE
{% extends "base.html" %}
{% load markup_textile %}
{% block title %}{{ page.title }} {% endblock %}
{% b...
SERIALIZER
• Convert objects into strings
for easy transport and data
exchange
Decorators
Router
Middleware
Controller (a....
APPLICATION ARCHITECTURE
Django ExtJS
Decorators
Router
Middleware
Controller (a.k.aView)
Template / Serializer
Middleware...
FRONTEND (EXTJS)
CODE STRUCTURE
Model
Controller
View
Store
HTTP
request
HTTP
response
event
view
update
VIEW
Model
Controller
View
Store
HTTP
request
HTTP
response
event
view
update
• Sets up visible GUI widgets
• Binds collec...
VIEW
EXAMPLE
Ext.create('Ext.grid.Panel', {
title: 'People',
alias : 'widget.peoplepanel',
store: Ext.data.StoreManager.lo...
VIEW
EXAMPLE
CONTROLLER
• Main application logic
• Sets up event listeners
• Interacts with models
• ...or sends custom requests to
the...
CONTROLLER
EXAMPLE
Ext.define('MyApp.controller.People', {
extend:'Ext.app.Controller',
init: function() {
this.control({
...
MODEL
• Represents API data as
JavaScript objects
• Client-side representation of
server-side models
• Performs data valid...
MODEL
EXAMPLE
Ext.define('Person', {
extend: 'Ext.data.Model',
idProperty: 'id',
fields: [
{name: 'id', type: 'int'},
{nam...
STORE
• Communicates with the API
• Enables UI data binding
• Converts API responses to
instances of models
Model
Controll...
STORE
EXAMPLE
Ext.define('MyApp.store.Persons',{
extend : 'Ext.data.Store',
storeId:'peopleStore',
model: 'MyApp.model.Per...
HTTP API DESIGN
headers
body
method URL
headers
body
status
HTTP request HTTP response
Cookies
User-agent
Language prefs
G...
TRADITIONAL HTTP
request response
headers
URLGET
headers
HTML
200
AJAXIFIED WEBSITE
request response
headers
POST
parameters
POST URL
headers
partial
HTML
200
REST
request response
Resource GET PUT POST DELETE
Collection URI List elements Replace collection Create element Delete c...
SOAP / JSON-RPC
REMOTE PROCEDURE CALL
request response
headers
method-call
and parameters
in an XML or JSON
envelope
POST ...
EXTJS-STYLE API
request response
headers
function
arguments
JSON
method function URI
headers
response
object
JSON
200
Demo
READY... SET... DEPLOY!!!
Gunicorn
Nginx
Django
PostgreSQL
Virtualenv
SCALING
Step 1: Optimize your code
• optimize number of DB queries
• rewrite slower ORM queries as
native SQL (stored proc...
SCALING
Step 2:Add Redis
• in memory key-value store
• as session store
• as cache backend
Gunicorn
Nginx
Django
Redis
Pos...
SCALING
Step 3:Add Varnish
• in-memory cache
• speed up serving static files
Gunicorn
Nginx
Django
Redis
PostgreSQL
Virtual...
SCALING
Step 4:
Split up your
servers
Gunicorn
Nginx
Django
Virtualenv
Varnish
Nginx
Static files
Varnish
PostgreSQLRedis
B...
SCALING
Step 5:
Clusters
PostgreSQL
Redis
App-node cluster
DB ClusterRedis cluster
(coming soon)
Gunicorn
Nginx
Django
Vir...
Thank you
http://michal.karzynski.pl
Scalable web application architecture
Upcoming SlideShare
Loading in...5
×

Scalable web application architecture

17,449

Published on

Talk describing a web application architecture based on Django and ExtJS. Presentation focuses on code organisation, deployment and scaling.

Published in: Technology
1 Comment
25 Likes
Statistics
Notes
  • Thank you ! your content its very good !
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
17,449
On Slideshare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
198
Comments
1
Likes
25
Embeds 0
No embeds

No notes for slide

Scalable web application architecture

  1. 1. SCALABLE WEB APPLICATION ARCHITECTURE (USING DJANGO + EXTJS) Michał Karzyński, PyWaw #27
  2. 2. TALK OUTLINE 1. Application architecture 2. Backend code organization (Django) 3. Frontend code organization (ExtJS) 4. Communication APIs 5. Deployment and Scaling
  3. 3. YOURSTRULY Michał Karzyński • project manager at Politechnika Gdańska • freelance developer and consultant • polyglot, currently: Python and JavaScript • web developer since 1996 • @postrational • http://michal.karzynski.pl
  4. 4. APPLICATION ARCHITECTURE Backend API Frontend Server Cloud Browser Python JSON JavaScript Django ExtJS I/O GUITransport
  5. 5. BACKEND (DJANGO) CODE STRUCTURE Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware HTTP Request HTTP Response Model (ORM)
  6. 6. MIDDLEWARE Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware Model (ORM) • Processes incoming requests and outgoing responses • Hooks in before or after the controller function • e.g. SessionMiddleware, CacheMiddleware
  7. 7. MIDDLEWARE EXAMPLE class SessionMiddleware(object): def process_request(self, request): engine = import_module(settings.SESSION_ENGINE) session_key = request.COOKIES.get( settings.SESSION_COOKIE_NAME, None) request.session = engine.SessionStore(session_key)
  8. 8. ROUTER • Maps URL patterns to controller function calls • Uses regular expressions • Can use positional and named parameters • Routes with default parameter values Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware Model (ORM)
  9. 9. ROUTER EXAMPLE urlpatterns = patterns('myapp.views', url(r'^$', home_page), # simple pattern url(r'^admin/', include(admin.site.urls)), # pattern inclusion url(r'^item/(w+)/(d+)/$', display_item), # positional params url(r'^info/(?P<slug>[-w]+)/$', display_page), # named params url(r'^info/$', display_page, { 'slug' : 'index' }), # default value )
  10. 10. DECORATORS • Wrap controller functions in additional logic • Can modify request object or cause a redirect • e.g. login_required, permission_required, cache_page Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware Model (ORM)
  11. 11. CONTROLLER (VIEW) • Main application logic • Functions receive an HTTP request and returns an HTTP response • Communicates with the model layer, session store, etc. Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware Model (ORM)
  12. 12. CONTROLLER (VIEW) EXAMPLE def display_institute(request, id): institute = get_object_or_404(Institute, pk=safeint(id)) return render_to_response('institute_display.html', { 'institute' : institute })
  13. 13. CONTROLLER (VIEW) EXAMPLE def save_offer(self, request): form_data = json_deserialize(request.read()) # Validate incoming offer data offer_form = OfferForm(form_data) validation_errors = offer_form.errors if not validation_errors: offer = offer_form.save() response = { 'success' : True, 'offer_url' : offer.get_absolute_url() } if validation_errors: for key in validation_errors.keys(): validation_errors[key] = " ".join(validation_errors[key]) response = { 'success' : False, 'errors' : validation_errors } return HttpResponse(json_serialize(response), mimetype='application/json')
  14. 14. MODEL (ORM) • Database abstraction layer • Converts DB rows into Python objects • Generates DB structure and lazy-loading SQL queries Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware Model (ORM)
  15. 15. MODEL (ORM) EXAMPLE class Person(models.Model): first_name = models.CharField(_('first name'), max_length=30, blank=True) last_name = models.CharField(_('last name'), max_length=30, blank=True) email = models.EmailField(_('email address'), blank=True) def get_full_name(self): """ Returns the first_name plus the last_name, with a space in between. """ full_name = '%s %s' % (self.first_name, self.last_name) return full_name.strip() Offer.objects.filter(deadline__date__gte=datetime.date.today()) .distinct().annotate(closest_deadline=Min('deadline__date')) .order_by('closest_deadline').select_related()
  16. 16. TEMPLATE • Generate HTML • Use Django’s templating language Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware Model (ORM)
  17. 17. TEMPLATE EXAMPLE {% extends "base.html" %} {% load markup_textile %} {% block title %}{{ page.title }} {% endblock %} {% block body %} <div class="page-header"> <h3>{{ page.title }}</h3> </div> <div class="span7"> <p> {{ page.body|textile }} </p> </div> {% if page.comments %} <div id="disqus_thread" class="span7"></div> {% endif %} {% endblock %}
  18. 18. SERIALIZER • Convert objects into strings for easy transport and data exchange Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware Model (ORM) return HttpResponse(json_serialize(data), mimetype='application/json')
  19. 19. APPLICATION ARCHITECTURE Django ExtJS Decorators Router Middleware Controller (a.k.aView) Template / Serializer Middleware Model Controller View Store Browser event click API request response view update Model (ORM)
  20. 20. FRONTEND (EXTJS) CODE STRUCTURE Model Controller View Store HTTP request HTTP response event view update
  21. 21. VIEW Model Controller View Store HTTP request HTTP response event view update • Sets up visible GUI widgets • Binds collection views to data stores
  22. 22. VIEW EXAMPLE Ext.create('Ext.grid.Panel', { title: 'People', alias : 'widget.peoplepanel', store: Ext.data.StoreManager.lookup('peopleStore'), columns: [ { text: 'First name', dataIndex: 'first_name' }, { text: 'Last name', dataIndex: 'last_name' }, { text: 'Email', dataIndex: 'email', flex: 1 } ], height: 200, width: 400, renderTo: Ext.getBody() });
  23. 23. VIEW EXAMPLE
  24. 24. CONTROLLER • Main application logic • Sets up event listeners • Interacts with models • ...or sends custom requests to the API, processes responses, updates the GUI Model Controller View Store HTTP request HTTP response event view update
  25. 25. CONTROLLER EXAMPLE Ext.define('MyApp.controller.People', { extend:'Ext.app.Controller', init: function() { this.control({ 'peoplepanel':{ itemclick : this.onPersonClicked } }); }, onPersonClicked: function(view, record, item, index, e, eOpts){ console.log('Person item clicked:' record.get('firstName')); record.set('was_clicked', true); record.save(); } });
  26. 26. MODEL • Represents API data as JavaScript objects • Client-side representation of server-side models • Performs data validation Model Controller View Store HTTP request HTTP response event view update
  27. 27. MODEL EXAMPLE Ext.define('Person', { extend: 'Ext.data.Model', idProperty: 'id', fields: [ {name: 'id', type: 'int'}, {name: 'first_name', type: 'string'}, {name: 'last_name', type: 'string'}, {name: 'age', type: 'int'}, {name: 'was_clicked',type: 'bool'} ], get_full_name: function() { return this.get('first_name') + ' ' + this.get('last_name'); } });
  28. 28. STORE • Communicates with the API • Enables UI data binding • Converts API responses to instances of models Model Controller View Store HTTP request HTTP response event view update
  29. 29. STORE EXAMPLE Ext.define('MyApp.store.Persons',{ extend : 'Ext.data.Store', storeId:'peopleStore', model: 'MyApp.model.Person' proxy: { type: 'ajax', api: { create : '/api/person/new', read : '/api/person/load', update : '/api/person/update', destroy : '/api/person/destroy' }, reader: { type: 'json', root: 'person', successProperty: 'success' } } });
  30. 30. HTTP API DESIGN headers body method URL headers body status HTTP request HTTP response Cookies User-agent Language prefs GET, POST, PUT, DELETE, etc. protocol, host port, resource, GET params uploaded files, JSON data, etc. Cookies Cache-control 200, 302, 404, 500, etc. HTML, JSON, binary data. etc.
  31. 31. TRADITIONAL HTTP request response headers URLGET headers HTML 200
  32. 32. AJAXIFIED WEBSITE request response headers POST parameters POST URL headers partial HTML 200
  33. 33. REST request response Resource GET PUT POST DELETE Collection URI List elements Replace collection Create element Delete collection Element URI Retrieve element Replace or create Delete element headers object JSON method resource headers object JSON 200, 201, 400, 409, etc.
  34. 34. SOAP / JSON-RPC REMOTE PROCEDURE CALL request response headers method-call and parameters in an XML or JSON envelope POST API endpoint headers method response object in an XML or JSON envelope 200
  35. 35. EXTJS-STYLE API request response headers function arguments JSON method function URI headers response object JSON 200
  36. 36. Demo
  37. 37. READY... SET... DEPLOY!!! Gunicorn Nginx Django PostgreSQL Virtualenv
  38. 38. SCALING Step 1: Optimize your code • optimize number of DB queries • rewrite slower ORM queries as native SQL (stored procedures) • use cacheing Gunicorn Nginx Django PostgreSQL Virtualenv
  39. 39. SCALING Step 2:Add Redis • in memory key-value store • as session store • as cache backend Gunicorn Nginx Django Redis PostgreSQL Virtualenv
  40. 40. SCALING Step 3:Add Varnish • in-memory cache • speed up serving static files Gunicorn Nginx Django Redis PostgreSQL Virtualenv Varnish
  41. 41. SCALING Step 4: Split up your servers Gunicorn Nginx Django Virtualenv Varnish Nginx Static files Varnish PostgreSQLRedis Backend Frontend DBCache
  42. 42. SCALING Step 5: Clusters PostgreSQL Redis App-node cluster DB ClusterRedis cluster (coming soon) Gunicorn Nginx Django Virtualenv Varnish Gunicorn Nginx Django Virtualenv Nginx Static files Varnish Nginx Static files Static files cluster or use a CDN Redis PostgreSQL
  43. 43. Thank you http://michal.karzynski.pl
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×