Evolving Archetecture
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Evolving Archetecture

on

  • 8,491 views

...


Looking at historic, current and evolving approaches, I will take you through from how we used to 'live' edit on one server with HTML in the code; to implementing Template Toolkit and 'front end / back end' servers; to the addition of version control; all the way through to distributed caching, file systems and processing (aka Six Apart worship) with 15+ servers.

Statistics

Views

Total Views
8,491
Views on SlideShare
8,481
Embed Views
10

Actions

Likes
6
Downloads
189
Comments
0

3 Embeds 10

http://www.slideshare.net 8
http://www.linkedin.com 1
https://www.linkedin.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Evolving Archetecture Presentation Transcript

  • 1. Evolving architecture make development easy and your site faster Leo Lapworth @ YAPC::EU 2007 http://leo.cuckoo.org/
  • 2. Covering
  • 3. Covering How running websites has evolved
  • 4. Covering How running websites has evolved Servers improved
  • 5. Covering How running websites has evolved Servers improved Architecture improved
  • 6. Covering How running websites has evolved Servers improved Architecture improved Development tools improved
  • 7. Covering How running websites has evolved Servers improved Architecture improved Development tools improved Code (and code libraries) got better
  • 8. but not...
  • 9. but not... Covering anything in depth
  • 10. Concepts stolen from
  • 11. Concepts stolen from Mod_perl guide
  • 12. Concepts stolen from Mod_perl guide Léon Brocard
  • 13. Concepts stolen from Mod_perl guide Léon Brocard Six Apart
  • 14. Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo!
  • 15. Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo! + many other people...
  • 16. Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo! + many other people... ... this is social Perl
  • 17. Templates
  • 18. Templates An example of how things have improved...
  • 19. Recognise this?
  • 20. Recognise this? use CGI; $q = CGI->new(); $name = $q->param('name'); print $q->header('text/html'); $link = 'http://foo.com/'; print quot;<html><head><title>HTML and code together</title></head><body>quot;; print quot;<a href=quot;$link/$namequot;>quot;; print quot;Click here $name</a><br />nquot;; print quot;<table>quot; . join(quot;nquot;, map { quot;<tr><td>$_</td></tr>quot; } qw(this is hard to maintain)); print quot;</table>quot;;
  • 21. How about...
  • 22. How about... Now, was it cellspacing, or cellpadding I need for ie4.02?
  • 23. and...
  • 24. and... I'm a developer, not a designer. I shouldn't have to know this £@$%!
  • 25. Use templates...
  • 26. Use templates... use Template;
  • 27. Code, not HTML
  • 28. Code, not HTML use CGI; use Template; $q = CGI->new(); print $q->header('text/html'); my %vals = ( name = $q->param('name'), ); $template = Template->new({ INCLUDE => '/path/' }); $template->process('hello.html',%vals);
  • 29. I just code, I have clean code and I don't have to know HTML or browser compatibility issues
  • 30. Simple changes make a huge difference
  • 31. Servers
  • 32. Single server
  • 33. Single server Internet Web server (static & dynamic) All editing Database server
  • 34. Lots of...
  • 35. and... I thought I only changed one thing, but it's still broken
  • 36. 1 live server 1 development server
  • 37. 1 live server 1 development server Internet Live server Development server Web server Web server All editing (static & dynamic) (static & dynamic) Database server Database server
  • 38. Does not solve: What did I change last? Who changed what / when?
  • 39. You need...
  • 40. You need...
  • 41. Version control & docs
  • 42. Version control & docs Use SVK with Subversion
  • 43. Version control & docs Use SVK with Subversion
  • 44. Version control & docs Use SVK with Subversion Clkao - SVK
  • 45. Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) Clkao - SVK
  • 46. Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki Clkao - SVK
  • 47. Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki - task tracker Clkao - SVK
  • 48. Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki - task tracker - web interface to Subversion Clkao - SVK
  • 49. Tests - all you need on one slide
  • 50. Tests - all you need on one slide Write tests, write tests often
  • 51. Tests - all you need on one slide Write tests, write tests often Run tests, run tests often
  • 52. Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good
  • 53. Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good Found a bug? - write the test that replicates it. You know it's fixed when your test passes
  • 54. Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good Found a bug? - write the test that replicates it. You know it's fixed when your test passes Use Test::More, and then test some more!
  • 55. Making your site faster
  • 56. mod_perl
  • 57. mod_perl Code caching (loaded/compiled once)
  • 58. mod_perl Code caching (loaded/compiled once) Server only spends time running the code
  • 59. mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code
  • 60. mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net)
  • 61. mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net) Set it high
  • 62. mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net) Set it high Check it
  • 63. Front/back end split
  • 64. Front/back end split Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • 65. or put another way...
  • 66. or put another way... Internet slow and demanding kids (users)
  • 67. or put another way... Canteen Internet Front end slow and Dinner-ladies - good at demanding serving precooked kids (static) food. Can order (users) from à la carte chefs and then serve
  • 68. or put another way... Canteen Kitchen Internet Front end Back end slow and Dinner-ladies - good at Chefs - good at demanding serving precooked making à la carte kids (static) food. Can order (dynamic) food (users) from à la carte chefs and then serve
  • 69. Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • 70. Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • 71. Speed up the frequent...
  • 72. Speed up the frequent... Are you re-creating the same objects/data structures again and again?
  • 73. Speed up the frequent... Are you re-creating the same objects/data structures again and again? Can this information persist for a period of time?
  • 74. Add caching
  • 75. Add caching Persisting data for a period of time
  • 76. Add caching Persisting data for a period of time Simple to implement:
  • 77. Add caching Persisting data for a period of time Simple to implement: Does cache exist?
  • 78. Add caching Persisting data for a period of time Simple to implement: Does cache exist? YES -> retrieve it and return it
  • 79. Add caching Persisting data for a period of time Simple to implement: Does cache exist? YES -> retrieve it and return it NO -> create data/object, store in cache and return it
  • 80. Things you might cache
  • 81. Things you might cache Search result sets (a list of ids)
  • 82. Things you might cache Search result sets (a list of ids) Individual items
  • 83. Things you might cache Search result sets (a list of ids) Individual items Look-ups of information from a database
  • 84. Things you might cache Search result sets (a list of ids) Individual items Look-ups of information from a database Fetching of information from external sources
  • 85. Centralise
  • 86. Centralise Put caching methods in one package
  • 87. Centralise Put caching methods in one package Put any generic methods in a package
  • 88. We now have a cache Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to Cache (dynamic) application server) Database server Development server
  • 89. We now have a cache Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to Cache Cache (dynamic) application server) Database server Development server
  • 90. Adding in the cache...
  • 91. Adding in the cache... Internet Kids
  • 92. Adding in the cache... Internet Front end Kids Dinner-ladies
  • 93. Adding in the cache... Internet Back end Front end Kids Chefs Dinner-ladies
  • 94. Adding in the cache... Cache Internet Back end Front end Magic Kids Chefs Dinner-ladies freezer
  • 95. Coping with more traffic
  • 96. Coping with more traffic Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Back end (application server) Web server (dynamic)
  • 97. Coping with more traffic Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Back end (application server) Web server (dynamic)
  • 98. That's two separate disks
  • 99. That's two separate disks Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server Cache server) (dynamic) Back end (application server) Web server Cache (dynamic)
  • 100. Second kitchen, two freezers...
  • 101. Second kitchen, two freezers... Internet Kids
  • 102. Second kitchen, two freezers... Internet Front end Kids Dinner-ladies
  • 103. Second kitchen, two freezers... Kitchen 1 Kitchen 2 Internet Front end Back end Kids Dinner-ladies Chefs
  • 104. Second kitchen, two freezers... Kitchen 1 Kitchen 2 Internet Front end Back end Cache Kids Dinner-ladies Chefs Freezers
  • 105. We want a cache across servers...
  • 106. We want a cache across servers... Enter Memcache
  • 107. We want a cache across servers... Enter Memcache See Léon Brocard's talk at 16:20 for the details
  • 108. Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Cache Back end (application server) Web server (dynamic)
  • 109. Shared freezer
  • 110. Shared freezer Internet Kids
  • 111. Shared freezer Internet Front end Kids Dinner-ladies
  • 112. Shared freezer Kitchen 1 Kitchen 2 Internet Front end Kids Dinner-ladies Back end Chefs
  • 113. Shared freezer Kitchen 1 Kitchen 2 Internet Front end Kids Dinner-ladies Back end Cache Chefs Freezer
  • 114. Maintaining servers
  • 115. Maintaining servers Keep all servers identical where possible
  • 116. Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs)
  • 117. Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs) Centralise your crontabs and configuration files (and put in version control)
  • 118. Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs) Centralise your crontabs and configuration files (and put in version control) Deploy with one script: rsync + ssh + apt-get update & apt-get upgrade (or your OS equivalent) to each server
  • 119. Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • 120. Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • 121. Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • 122. Waiters: cheaper, faster and less demanding
  • 123. Waiters: cheaper, faster and less demanding Internet Kids
  • 124. Waiters: cheaper, faster and less demanding Front end Internet Waiters - good at serving Kids quickly and dealing with annoying customers or Kitchens
  • 125. Waiters: cheaper, faster and less demanding Kitchen 1 Kitchen 2 Front end Internet Waiters - good at serving Kids Back end quickly and dealing with annoying customers or Chefs - simple and complex meals (static and dynamic) Kitchens
  • 126. Waiters: cheaper, faster and less demanding Kitchen 1 Kitchen 2 Front end Internet Waiters - good at serving Kids Back end quickly and dealing with Cache annoying customers or Chefs - simple and complex meals Freezer (static and dynamic) Kitchens
  • 127. More tips for faster user experience
  • 128. mod_gzip
  • 129. mod_gzip HTML/css/xml
  • 130. mod_gzip HTML/css/xml
  • 131. mod_gzip gzip HTML/css/xml
  • 132. mod_gzip gzip HTML/css/xml
  • 133. mod_gzip gzip HTML/css/xml Users
  • 134. mod_gzip gzip HTML/css/xml Users If a browser support compressions... compress (javascript can be tricky - at least minify)
  • 135. Cache headers (expiry)
  • 136. Cache headers (expiry)
  • 137. Cache headers (expiry) Let web browsers know how long to cache
  • 138. Cache somethings forever
  • 139. Cache somethings forever /includes/js/<VERSION>/common.js
  • 140. Cache somethings forever /includes/js/<VERSION>/common.js Ensures user has version which matches HTML (client could have cache of old HTML)
  • 141. Cache somethings forever /includes/js/<VERSION>/common.js Ensures user has version which matches HTML (client could have cache of old HTML) Use include file to update all pages
  • 142. Handling images: MogileFS
  • 143. Handling images: MogileFS Store images by group (replication levels)
  • 144. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers
  • 145. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server
  • 146. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user
  • 147. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user Automatic resilience
  • 148. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user Automatic resilience
  • 149. Basic rules to consider
  • 150. Basic rules to consider Centralise (code/templates/configuration/ deployment etc)
  • 151. Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them
  • 152. Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great
  • 153. Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great Keep is simple (perlbal especially)
  • 154. Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great Keep is simple (perlbal especially) Sometimes it's just easier to buy another server (this is not always true)
  • 155. We covered...
  • 156. We covered... perlbal
  • 157. We covered... perlbal memcached
  • 158. We covered... perlbal memcached svk (subversion)
  • 159. We covered... perlbal memcached svk (subversion) trac
  • 160. We covered... perlbal memcached svk (subversion) trac rsync
  • 161. We covered... perlbal memcached svk (subversion) trac rsync gzip
  • 162. We covered... perlbal mogileFS memcached svk (subversion) trac rsync gzip
  • 163. We covered... perlbal mogileFS server architecture memcached svk (subversion) trac rsync gzip
  • 164. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) trac rsync gzip
  • 165. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac rsync gzip
  • 166. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync gzip
  • 167. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync HTTP Cache headers gzip
  • 168. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync HTTP Cache headers gzip and...
  • 169. ...how a kitchen should run
  • 170. ...how a kitchen should run
  • 171. ...how a kitchen should run Any questions?
  • 172. Evolving architecture http://leo.cuckoo.org/projects/ea/ Leo Lapworth http://leo.cuckoo.org/