Evolving Archetecture
Upcoming SlideShare
Loading in...5
×
 

Evolving Archetecture

on

  • 8,445 views

...


Looking at historic, current and evolving approaches, I will take you through from how we used to 'live' edit on one server with HTML in the code; to implementing Template Toolkit and 'front end / back end' servers; to the addition of version control; all the way through to distributed caching, file systems and processing (aka Six Apart worship) with 15+ servers.

Statistics

Views

Total Views
8,445
Views on SlideShare
8,435
Embed Views
10

Actions

Likes
6
Downloads
189
Comments
0

3 Embeds 10

http://www.slideshare.net 8
http://www.linkedin.com 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Evolving Archetecture Evolving Archetecture Presentation Transcript

  • Evolving architecture make development easy and your site faster Leo Lapworth @ YAPC::EU 2007 http://leo.cuckoo.org/
  • Covering
  • Covering How running websites has evolved
  • Covering How running websites has evolved Servers improved
  • Covering How running websites has evolved Servers improved Architecture improved
  • Covering How running websites has evolved Servers improved Architecture improved Development tools improved
  • Covering How running websites has evolved Servers improved Architecture improved Development tools improved Code (and code libraries) got better
  • but not...
  • but not... Covering anything in depth
  • Concepts stolen from
  • Concepts stolen from Mod_perl guide
  • Concepts stolen from Mod_perl guide Léon Brocard
  • Concepts stolen from Mod_perl guide Léon Brocard Six Apart
  • Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo!
  • Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo! + many other people...
  • Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo! + many other people... ... this is social Perl
  • Templates
  • Templates An example of how things have improved...
  • Recognise this?
  • Recognise this? use CGI; $q = CGI->new(); $name = $q->param('name'); print $q->header('text/html'); $link = 'http://foo.com/'; print quot;<html><head><title>HTML and code together</title></head><body>quot;; print quot;<a href=quot;$link/$namequot;>quot;; print quot;Click here $name</a><br />nquot;; print quot;<table>quot; . join(quot;nquot;, map { quot;<tr><td>$_</td></tr>quot; } qw(this is hard to maintain)); print quot;</table>quot;;
  • How about...
  • How about... Now, was it cellspacing, or cellpadding I need for ie4.02?
  • and...
  • and... I'm a developer, not a designer. I shouldn't have to know this £@$%!
  • Use templates...
  • Use templates... use Template;
  • Code, not HTML
  • Code, not HTML use CGI; use Template; $q = CGI->new(); print $q->header('text/html'); my %vals = ( name = $q->param('name'), ); $template = Template->new({ INCLUDE => '/path/' }); $template->process('hello.html',%vals);
  • I just code, I have clean code and I don't have to know HTML or browser compatibility issues
  • Simple changes make a huge difference
  • Servers
  • Single server
  • Single server Internet Web server (static & dynamic) All editing Database server
  • Lots of...
  • and... I thought I only changed one thing, but it's still broken
  • 1 live server 1 development server
  • 1 live server 1 development server Internet Live server Development server Web server Web server All editing (static & dynamic) (static & dynamic) Database server Database server
  • Does not solve: What did I change last? Who changed what / when?
  • You need...
  • You need...
  • Version control & docs
  • Version control & docs Use SVK with Subversion
  • Version control & docs Use SVK with Subversion
  • Version control & docs Use SVK with Subversion Clkao - SVK
  • Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) Clkao - SVK
  • Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki Clkao - SVK
  • Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki - task tracker Clkao - SVK
  • Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki - task tracker - web interface to Subversion Clkao - SVK
  • Tests - all you need on one slide
  • Tests - all you need on one slide Write tests, write tests often
  • Tests - all you need on one slide Write tests, write tests often Run tests, run tests often
  • Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good
  • Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good Found a bug? - write the test that replicates it. You know it's fixed when your test passes
  • Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good Found a bug? - write the test that replicates it. You know it's fixed when your test passes Use Test::More, and then test some more!
  • Making your site faster
  • mod_perl
  • mod_perl Code caching (loaded/compiled once)
  • mod_perl Code caching (loaded/compiled once) Server only spends time running the code
  • mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code
  • mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net)
  • mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net) Set it high
  • mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net) Set it high Check it
  • Front/back end split
  • Front/back end split Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • or put another way...
  • or put another way... Internet slow and demanding kids (users)
  • or put another way... Canteen Internet Front end slow and Dinner-ladies - good at demanding serving precooked kids (static) food. Can order (users) from à la carte chefs and then serve
  • or put another way... Canteen Kitchen Internet Front end Back end slow and Dinner-ladies - good at Chefs - good at demanding serving precooked making à la carte kids (static) food. Can order (dynamic) food (users) from à la carte chefs and then serve
  • Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • Speed up the frequent...
  • Speed up the frequent... Are you re-creating the same objects/data structures again and again?
  • Speed up the frequent... Are you re-creating the same objects/data structures again and again? Can this information persist for a period of time?
  • Add caching
  • Add caching Persisting data for a period of time
  • Add caching Persisting data for a period of time Simple to implement:
  • Add caching Persisting data for a period of time Simple to implement: Does cache exist?
  • Add caching Persisting data for a period of time Simple to implement: Does cache exist? YES -> retrieve it and return it
  • Add caching Persisting data for a period of time Simple to implement: Does cache exist? YES -> retrieve it and return it NO -> create data/object, store in cache and return it
  • Things you might cache
  • Things you might cache Search result sets (a list of ids)
  • Things you might cache Search result sets (a list of ids) Individual items
  • Things you might cache Search result sets (a list of ids) Individual items Look-ups of information from a database
  • Things you might cache Search result sets (a list of ids) Individual items Look-ups of information from a database Fetching of information from external sources
  • Centralise
  • Centralise Put caching methods in one package
  • Centralise Put caching methods in one package Put any generic methods in a package
  • We now have a cache Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to Cache (dynamic) application server) Database server Development server
  • We now have a cache Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to Cache Cache (dynamic) application server) Database server Development server
  • Adding in the cache...
  • Adding in the cache... Internet Kids
  • Adding in the cache... Internet Front end Kids Dinner-ladies
  • Adding in the cache... Internet Back end Front end Kids Chefs Dinner-ladies
  • Adding in the cache... Cache Internet Back end Front end Magic Kids Chefs Dinner-ladies freezer
  • Coping with more traffic
  • Coping with more traffic Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Back end (application server) Web server (dynamic)
  • Coping with more traffic Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Back end (application server) Web server (dynamic)
  • That's two separate disks
  • That's two separate disks Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server Cache server) (dynamic) Back end (application server) Web server Cache (dynamic)
  • Second kitchen, two freezers...
  • Second kitchen, two freezers... Internet Kids
  • Second kitchen, two freezers... Internet Front end Kids Dinner-ladies
  • Second kitchen, two freezers... Kitchen 1 Kitchen 2 Internet Front end Back end Kids Dinner-ladies Chefs
  • Second kitchen, two freezers... Kitchen 1 Kitchen 2 Internet Front end Back end Cache Kids Dinner-ladies Chefs Freezers
  • We want a cache across servers...
  • We want a cache across servers... Enter Memcache
  • We want a cache across servers... Enter Memcache See Léon Brocard's talk at 16:20 for the details
  • Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Cache Back end (application server) Web server (dynamic)
  • Shared freezer
  • Shared freezer Internet Kids
  • Shared freezer Internet Front end Kids Dinner-ladies
  • Shared freezer Kitchen 1 Kitchen 2 Internet Front end Kids Dinner-ladies Back end Chefs
  • Shared freezer Kitchen 1 Kitchen 2 Internet Front end Kids Dinner-ladies Back end Cache Chefs Freezer
  • Maintaining servers
  • Maintaining servers Keep all servers identical where possible
  • Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs)
  • Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs) Centralise your crontabs and configuration files (and put in version control)
  • Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs) Centralise your crontabs and configuration files (and put in version control) Deploy with one script: rsync + ssh + apt-get update & apt-get upgrade (or your OS equivalent) to each server
  • Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • Waiters: cheaper, faster and less demanding
  • Waiters: cheaper, faster and less demanding Internet Kids
  • Waiters: cheaper, faster and less demanding Front end Internet Waiters - good at serving Kids quickly and dealing with annoying customers or Kitchens
  • Waiters: cheaper, faster and less demanding Kitchen 1 Kitchen 2 Front end Internet Waiters - good at serving Kids Back end quickly and dealing with annoying customers or Chefs - simple and complex meals (static and dynamic) Kitchens
  • Waiters: cheaper, faster and less demanding Kitchen 1 Kitchen 2 Front end Internet Waiters - good at serving Kids Back end quickly and dealing with Cache annoying customers or Chefs - simple and complex meals Freezer (static and dynamic) Kitchens
  • More tips for faster user experience
  • mod_gzip
  • mod_gzip HTML/css/xml
  • mod_gzip HTML/css/xml
  • mod_gzip gzip HTML/css/xml
  • mod_gzip gzip HTML/css/xml
  • mod_gzip gzip HTML/css/xml Users
  • mod_gzip gzip HTML/css/xml Users If a browser support compressions... compress (javascript can be tricky - at least minify)
  • Cache headers (expiry)
  • Cache headers (expiry)
  • Cache headers (expiry) Let web browsers know how long to cache
  • Cache somethings forever
  • Cache somethings forever /includes/js/<VERSION>/common.js
  • Cache somethings forever /includes/js/<VERSION>/common.js Ensures user has version which matches HTML (client could have cache of old HTML)
  • Cache somethings forever /includes/js/<VERSION>/common.js Ensures user has version which matches HTML (client could have cache of old HTML) Use include file to update all pages
  • Handling images: MogileFS
  • Handling images: MogileFS Store images by group (replication levels)
  • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers
  • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server
  • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user
  • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user Automatic resilience
  • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user Automatic resilience
  • Basic rules to consider
  • Basic rules to consider Centralise (code/templates/configuration/ deployment etc)
  • Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them
  • Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great
  • Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great Keep is simple (perlbal especially)
  • Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great Keep is simple (perlbal especially) Sometimes it's just easier to buy another server (this is not always true)
  • We covered...
  • We covered... perlbal
  • We covered... perlbal memcached
  • We covered... perlbal memcached svk (subversion)
  • We covered... perlbal memcached svk (subversion) trac
  • We covered... perlbal memcached svk (subversion) trac rsync
  • We covered... perlbal memcached svk (subversion) trac rsync gzip
  • We covered... perlbal mogileFS memcached svk (subversion) trac rsync gzip
  • We covered... perlbal mogileFS server architecture memcached svk (subversion) trac rsync gzip
  • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) trac rsync gzip
  • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac rsync gzip
  • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync gzip
  • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync HTTP Cache headers gzip
  • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync HTTP Cache headers gzip and...
  • ...how a kitchen should run
  • ...how a kitchen should run
  • ...how a kitchen should run Any questions?
  • Evolving architecture http://leo.cuckoo.org/projects/ea/ Leo Lapworth http://leo.cuckoo.org/