Evolving Archetecture
Upcoming SlideShare
Loading in...5
×
 

Evolving Archetecture

on

  • 8,405 views

...


Looking at historic, current and evolving approaches, I will take you through from how we used to 'live' edit on one server with HTML in the code; to implementing Template Toolkit and 'front end / back end' servers; to the addition of version control; all the way through to distributed caching, file systems and processing (aka Six Apart worship) with 15+ servers.

Statistics

Views

Total Views
8,405
Views on SlideShare
8,395
Embed Views
10

Actions

Likes
6
Downloads
189
Comments
0

3 Embeds 10

http://www.slideshare.net 8
http://www.linkedin.com 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Evolving Archetecture Evolving Archetecture Presentation Transcript

    • Evolving architecture make development easy and your site faster Leo Lapworth @ YAPC::EU 2007 http://leo.cuckoo.org/
    • Covering
    • Covering How running websites has evolved
    • Covering How running websites has evolved Servers improved
    • Covering How running websites has evolved Servers improved Architecture improved
    • Covering How running websites has evolved Servers improved Architecture improved Development tools improved
    • Covering How running websites has evolved Servers improved Architecture improved Development tools improved Code (and code libraries) got better
    • but not...
    • but not... Covering anything in depth
    • Concepts stolen from
    • Concepts stolen from Mod_perl guide
    • Concepts stolen from Mod_perl guide Léon Brocard
    • Concepts stolen from Mod_perl guide Léon Brocard Six Apart
    • Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo!
    • Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo! + many other people...
    • Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo! + many other people... ... this is social Perl
    • Templates
    • Templates An example of how things have improved...
    • Recognise this?
    • Recognise this? use CGI; $q = CGI->new(); $name = $q->param('name'); print $q->header('text/html'); $link = 'http://foo.com/'; print quot;<html><head><title>HTML and code together</title></head><body>quot;; print quot;<a href=quot;$link/$namequot;>quot;; print quot;Click here $name</a><br />nquot;; print quot;<table>quot; . join(quot;nquot;, map { quot;<tr><td>$_</td></tr>quot; } qw(this is hard to maintain)); print quot;</table>quot;;
    • How about...
    • How about... Now, was it cellspacing, or cellpadding I need for ie4.02?
    • and...
    • and... I'm a developer, not a designer. I shouldn't have to know this £@$%!
    • Use templates...
    • Use templates... use Template;
    • Code, not HTML
    • Code, not HTML use CGI; use Template; $q = CGI->new(); print $q->header('text/html'); my %vals = ( name = $q->param('name'), ); $template = Template->new({ INCLUDE => '/path/' }); $template->process('hello.html',%vals);
    • I just code, I have clean code and I don't have to know HTML or browser compatibility issues
    • Simple changes make a huge difference
    • Servers
    • Single server
    • Single server Internet Web server (static & dynamic) All editing Database server
    • Lots of...
    • and... I thought I only changed one thing, but it's still broken
    • 1 live server 1 development server
    • 1 live server 1 development server Internet Live server Development server Web server Web server All editing (static & dynamic) (static & dynamic) Database server Database server
    • Does not solve: What did I change last? Who changed what / when?
    • You need...
    • You need...
    • Version control & docs
    • Version control & docs Use SVK with Subversion
    • Version control & docs Use SVK with Subversion
    • Version control & docs Use SVK with Subversion Clkao - SVK
    • Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) Clkao - SVK
    • Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki Clkao - SVK
    • Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki - task tracker Clkao - SVK
    • Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki - task tracker - web interface to Subversion Clkao - SVK
    • Tests - all you need on one slide
    • Tests - all you need on one slide Write tests, write tests often
    • Tests - all you need on one slide Write tests, write tests often Run tests, run tests often
    • Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good
    • Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good Found a bug? - write the test that replicates it. You know it's fixed when your test passes
    • Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good Found a bug? - write the test that replicates it. You know it's fixed when your test passes Use Test::More, and then test some more!
    • Making your site faster
    • mod_perl
    • mod_perl Code caching (loaded/compiled once)
    • mod_perl Code caching (loaded/compiled once) Server only spends time running the code
    • mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code
    • mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net)
    • mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net) Set it high
    • mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net) Set it high Check it
    • Front/back end split
    • Front/back end split Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
    • or put another way...
    • or put another way... Internet slow and demanding kids (users)
    • or put another way... Canteen Internet Front end slow and Dinner-ladies - good at demanding serving precooked kids (static) food. Can order (users) from à la carte chefs and then serve
    • or put another way... Canteen Kitchen Internet Front end Back end slow and Dinner-ladies - good at Chefs - good at demanding serving precooked making à la carte kids (static) food. Can order (dynamic) food (users) from à la carte chefs and then serve
    • Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
    • Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
    • Speed up the frequent...
    • Speed up the frequent... Are you re-creating the same objects/data structures again and again?
    • Speed up the frequent... Are you re-creating the same objects/data structures again and again? Can this information persist for a period of time?
    • Add caching
    • Add caching Persisting data for a period of time
    • Add caching Persisting data for a period of time Simple to implement:
    • Add caching Persisting data for a period of time Simple to implement: Does cache exist?
    • Add caching Persisting data for a period of time Simple to implement: Does cache exist? YES -> retrieve it and return it
    • Add caching Persisting data for a period of time Simple to implement: Does cache exist? YES -> retrieve it and return it NO -> create data/object, store in cache and return it
    • Things you might cache
    • Things you might cache Search result sets (a list of ids)
    • Things you might cache Search result sets (a list of ids) Individual items
    • Things you might cache Search result sets (a list of ids) Individual items Look-ups of information from a database
    • Things you might cache Search result sets (a list of ids) Individual items Look-ups of information from a database Fetching of information from external sources
    • Centralise
    • Centralise Put caching methods in one package
    • Centralise Put caching methods in one package Put any generic methods in a package
    • We now have a cache Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to Cache (dynamic) application server) Database server Development server
    • We now have a cache Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to Cache Cache (dynamic) application server) Database server Development server
    • Adding in the cache...
    • Adding in the cache... Internet Kids
    • Adding in the cache... Internet Front end Kids Dinner-ladies
    • Adding in the cache... Internet Back end Front end Kids Chefs Dinner-ladies
    • Adding in the cache... Cache Internet Back end Front end Magic Kids Chefs Dinner-ladies freezer
    • Coping with more traffic
    • Coping with more traffic Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Back end (application server) Web server (dynamic)
    • Coping with more traffic Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Back end (application server) Web server (dynamic)
    • That's two separate disks
    • That's two separate disks Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server Cache server) (dynamic) Back end (application server) Web server Cache (dynamic)
    • Second kitchen, two freezers...
    • Second kitchen, two freezers... Internet Kids
    • Second kitchen, two freezers... Internet Front end Kids Dinner-ladies
    • Second kitchen, two freezers... Kitchen 1 Kitchen 2 Internet Front end Back end Kids Dinner-ladies Chefs
    • Second kitchen, two freezers... Kitchen 1 Kitchen 2 Internet Front end Back end Cache Kids Dinner-ladies Chefs Freezers
    • We want a cache across servers...
    • We want a cache across servers... Enter Memcache
    • We want a cache across servers... Enter Memcache See Léon Brocard's talk at 16:20 for the details
    • Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Cache Back end (application server) Web server (dynamic)
    • Shared freezer
    • Shared freezer Internet Kids
    • Shared freezer Internet Front end Kids Dinner-ladies
    • Shared freezer Kitchen 1 Kitchen 2 Internet Front end Kids Dinner-ladies Back end Chefs
    • Shared freezer Kitchen 1 Kitchen 2 Internet Front end Kids Dinner-ladies Back end Cache Chefs Freezer
    • Maintaining servers
    • Maintaining servers Keep all servers identical where possible
    • Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs)
    • Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs) Centralise your crontabs and configuration files (and put in version control)
    • Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs) Centralise your crontabs and configuration files (and put in version control) Deploy with one script: rsync + ssh + apt-get update & apt-get upgrade (or your OS equivalent) to each server
    • Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
    • Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
    • Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
    • Waiters: cheaper, faster and less demanding
    • Waiters: cheaper, faster and less demanding Internet Kids
    • Waiters: cheaper, faster and less demanding Front end Internet Waiters - good at serving Kids quickly and dealing with annoying customers or Kitchens
    • Waiters: cheaper, faster and less demanding Kitchen 1 Kitchen 2 Front end Internet Waiters - good at serving Kids Back end quickly and dealing with annoying customers or Chefs - simple and complex meals (static and dynamic) Kitchens
    • Waiters: cheaper, faster and less demanding Kitchen 1 Kitchen 2 Front end Internet Waiters - good at serving Kids Back end quickly and dealing with Cache annoying customers or Chefs - simple and complex meals Freezer (static and dynamic) Kitchens
    • More tips for faster user experience
    • mod_gzip
    • mod_gzip HTML/css/xml
    • mod_gzip HTML/css/xml
    • mod_gzip gzip HTML/css/xml
    • mod_gzip gzip HTML/css/xml
    • mod_gzip gzip HTML/css/xml Users
    • mod_gzip gzip HTML/css/xml Users If a browser support compressions... compress (javascript can be tricky - at least minify)
    • Cache headers (expiry)
    • Cache headers (expiry)
    • Cache headers (expiry) Let web browsers know how long to cache
    • Cache somethings forever
    • Cache somethings forever /includes/js/<VERSION>/common.js
    • Cache somethings forever /includes/js/<VERSION>/common.js Ensures user has version which matches HTML (client could have cache of old HTML)
    • Cache somethings forever /includes/js/<VERSION>/common.js Ensures user has version which matches HTML (client could have cache of old HTML) Use include file to update all pages
    • Handling images: MogileFS
    • Handling images: MogileFS Store images by group (replication levels)
    • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers
    • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server
    • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user
    • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user Automatic resilience
    • Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user Automatic resilience
    • Basic rules to consider
    • Basic rules to consider Centralise (code/templates/configuration/ deployment etc)
    • Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them
    • Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great
    • Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great Keep is simple (perlbal especially)
    • Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great Keep is simple (perlbal especially) Sometimes it's just easier to buy another server (this is not always true)
    • We covered...
    • We covered... perlbal
    • We covered... perlbal memcached
    • We covered... perlbal memcached svk (subversion)
    • We covered... perlbal memcached svk (subversion) trac
    • We covered... perlbal memcached svk (subversion) trac rsync
    • We covered... perlbal memcached svk (subversion) trac rsync gzip
    • We covered... perlbal mogileFS memcached svk (subversion) trac rsync gzip
    • We covered... perlbal mogileFS server architecture memcached svk (subversion) trac rsync gzip
    • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) trac rsync gzip
    • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac rsync gzip
    • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync gzip
    • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync HTTP Cache headers gzip
    • We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync HTTP Cache headers gzip and...
    • ...how a kitchen should run
    • ...how a kitchen should run
    • ...how a kitchen should run Any questions?
    • Evolving architecture http://leo.cuckoo.org/projects/ea/ Leo Lapworth http://leo.cuckoo.org/