Evolving Archetecture
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Evolving Archetecture

  • 8,525 views
Uploaded on

...


Looking at historic, current and evolving approaches, I will take you through from how we used to 'live' edit on one server with HTML in the code; to implementing Template Toolkit and 'front end / back end' servers; to the addition of version control; all the way through to distributed caching, file systems and processing (aka Six Apart worship) with 15+ servers.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,525
On Slideshare
8,515
From Embeds
10
Number of Embeds
3

Actions

Shares
Downloads
189
Comments
0
Likes
6

Embeds 10

http://www.slideshare.net 8
http://www.linkedin.com 1
https://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Evolving architecture make development easy and your site faster Leo Lapworth @ YAPC::EU 2007 http://leo.cuckoo.org/
  • 2. Covering
  • 3. Covering How running websites has evolved
  • 4. Covering How running websites has evolved Servers improved
  • 5. Covering How running websites has evolved Servers improved Architecture improved
  • 6. Covering How running websites has evolved Servers improved Architecture improved Development tools improved
  • 7. Covering How running websites has evolved Servers improved Architecture improved Development tools improved Code (and code libraries) got better
  • 8. but not...
  • 9. but not... Covering anything in depth
  • 10. Concepts stolen from
  • 11. Concepts stolen from Mod_perl guide
  • 12. Concepts stolen from Mod_perl guide Léon Brocard
  • 13. Concepts stolen from Mod_perl guide Léon Brocard Six Apart
  • 14. Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo!
  • 15. Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo! + many other people...
  • 16. Concepts stolen from Mod_perl guide Léon Brocard Six Apart Yahoo! + many other people... ... this is social Perl
  • 17. Templates
  • 18. Templates An example of how things have improved...
  • 19. Recognise this?
  • 20. Recognise this? use CGI; $q = CGI->new(); $name = $q->param('name'); print $q->header('text/html'); $link = 'http://foo.com/'; print quot;<html><head><title>HTML and code together</title></head><body>quot;; print quot;<a href=quot;$link/$namequot;>quot;; print quot;Click here $name</a><br />nquot;; print quot;<table>quot; . join(quot;nquot;, map { quot;<tr><td>$_</td></tr>quot; } qw(this is hard to maintain)); print quot;</table>quot;;
  • 21. How about...
  • 22. How about... Now, was it cellspacing, or cellpadding I need for ie4.02?
  • 23. and...
  • 24. and... I'm a developer, not a designer. I shouldn't have to know this £@$%!
  • 25. Use templates...
  • 26. Use templates... use Template;
  • 27. Code, not HTML
  • 28. Code, not HTML use CGI; use Template; $q = CGI->new(); print $q->header('text/html'); my %vals = ( name = $q->param('name'), ); $template = Template->new({ INCLUDE => '/path/' }); $template->process('hello.html',%vals);
  • 29. I just code, I have clean code and I don't have to know HTML or browser compatibility issues
  • 30. Simple changes make a huge difference
  • 31. Servers
  • 32. Single server
  • 33. Single server Internet Web server (static & dynamic) All editing Database server
  • 34. Lots of...
  • 35. and... I thought I only changed one thing, but it's still broken
  • 36. 1 live server 1 development server
  • 37. 1 live server 1 development server Internet Live server Development server Web server Web server All editing (static & dynamic) (static & dynamic) Database server Database server
  • 38. Does not solve: What did I change last? Who changed what / when?
  • 39. You need...
  • 40. You need...
  • 41. Version control & docs
  • 42. Version control & docs Use SVK with Subversion
  • 43. Version control & docs Use SVK with Subversion
  • 44. Version control & docs Use SVK with Subversion Clkao - SVK
  • 45. Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) Clkao - SVK
  • 46. Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki Clkao - SVK
  • 47. Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki - task tracker Clkao - SVK
  • 48. Version control & docs Use SVK with Subversion Use Trac (http://trac.edgewall.org) - wiki - task tracker - web interface to Subversion Clkao - SVK
  • 49. Tests - all you need on one slide
  • 50. Tests - all you need on one slide Write tests, write tests often
  • 51. Tests - all you need on one slide Write tests, write tests often Run tests, run tests often
  • 52. Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good
  • 53. Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good Found a bug? - write the test that replicates it. You know it's fixed when your test passes
  • 54. Tests - all you need on one slide Write tests, write tests often Run tests, run tests often Tests are good Found a bug? - write the test that replicates it. You know it's fixed when your test passes Use Test::More, and then test some more!
  • 55. Making your site faster
  • 56. mod_perl
  • 57. mod_perl Code caching (loaded/compiled once)
  • 58. mod_perl Code caching (loaded/compiled once) Server only spends time running the code
  • 59. mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code
  • 60. mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net)
  • 61. mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net) Set it high
  • 62. mod_perl Code caching (loaded/compiled once) Server only spends time running the code Apache::Registry for old/existing code Apache::SizeLimit (safety net) Set it high Check it
  • 63. Front/back end split
  • 64. Front/back end split Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • 65. or put another way...
  • 66. or put another way... Internet slow and demanding kids (users)
  • 67. or put another way... Canteen Internet Front end slow and Dinner-ladies - good at demanding serving precooked kids (static) food. Can order (users) from à la carte chefs and then serve
  • 68. or put another way... Canteen Kitchen Internet Front end Back end slow and Dinner-ladies - good at Chefs - good at demanding serving precooked making à la carte kids (static) food. Can order (dynamic) food (users) from à la carte chefs and then serve
  • 69. Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • 70. Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to (dynamic) application server) Database server Development server
  • 71. Speed up the frequent...
  • 72. Speed up the frequent... Are you re-creating the same objects/data structures again and again?
  • 73. Speed up the frequent... Are you re-creating the same objects/data structures again and again? Can this information persist for a period of time?
  • 74. Add caching
  • 75. Add caching Persisting data for a period of time
  • 76. Add caching Persisting data for a period of time Simple to implement:
  • 77. Add caching Persisting data for a period of time Simple to implement: Does cache exist?
  • 78. Add caching Persisting data for a period of time Simple to implement: Does cache exist? YES -> retrieve it and return it
  • 79. Add caching Persisting data for a period of time Simple to implement: Does cache exist? YES -> retrieve it and return it NO -> create data/object, store in cache and return it
  • 80. Things you might cache
  • 81. Things you might cache Search result sets (a list of ids)
  • 82. Things you might cache Search result sets (a list of ids) Individual items
  • 83. Things you might cache Search result sets (a list of ids) Individual items Look-ups of information from a database
  • 84. Things you might cache Search result sets (a list of ids) Individual items Look-ups of information from a database Fetching of information from external sources
  • 85. Centralise
  • 86. Centralise Put caching methods in one package
  • 87. Centralise Put caching methods in one package Put any generic methods in a package
  • 88. We now have a cache Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to Cache (dynamic) application server) Database server Development server
  • 89. We now have a cache Internet Slow Back end Front end (application server) Fast Web server Web server (static & proxy to Cache Cache (dynamic) application server) Database server Development server
  • 90. Adding in the cache...
  • 91. Adding in the cache... Internet Kids
  • 92. Adding in the cache... Internet Front end Kids Dinner-ladies
  • 93. Adding in the cache... Internet Back end Front end Kids Chefs Dinner-ladies
  • 94. Adding in the cache... Cache Internet Back end Front end Magic Kids Chefs Dinner-ladies freezer
  • 95. Coping with more traffic
  • 96. Coping with more traffic Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Back end (application server) Web server (dynamic)
  • 97. Coping with more traffic Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Back end (application server) Web server (dynamic)
  • 98. That's two separate disks
  • 99. That's two separate disks Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server Cache server) (dynamic) Back end (application server) Web server Cache (dynamic)
  • 100. Second kitchen, two freezers...
  • 101. Second kitchen, two freezers... Internet Kids
  • 102. Second kitchen, two freezers... Internet Front end Kids Dinner-ladies
  • 103. Second kitchen, two freezers... Kitchen 1 Kitchen 2 Internet Front end Back end Kids Dinner-ladies Chefs
  • 104. Second kitchen, two freezers... Kitchen 1 Kitchen 2 Internet Front end Back end Cache Kids Dinner-ladies Chefs Freezers
  • 105. We want a cache across servers...
  • 106. We want a cache across servers... Enter Memcache
  • 107. We want a cache across servers... Enter Memcache See Léon Brocard's talk at 16:20 for the details
  • 108. Development Internet Stage server server (in the office) Front end Back end Web server (application server) Database (static & proxy server to application Web server server) (dynamic) Cache Back end (application server) Web server (dynamic)
  • 109. Shared freezer
  • 110. Shared freezer Internet Kids
  • 111. Shared freezer Internet Front end Kids Dinner-ladies
  • 112. Shared freezer Kitchen 1 Kitchen 2 Internet Front end Kids Dinner-ladies Back end Chefs
  • 113. Shared freezer Kitchen 1 Kitchen 2 Internet Front end Kids Dinner-ladies Back end Cache Chefs Freezer
  • 114. Maintaining servers
  • 115. Maintaining servers Keep all servers identical where possible
  • 116. Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs)
  • 117. Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs) Centralise your crontabs and configuration files (and put in version control)
  • 118. Maintaining servers Keep all servers identical where possible Package your code (e.g. .debs) Centralise your crontabs and configuration files (and put in version control) Deploy with one script: rsync + ssh + apt-get update & apt-get upgrade (or your OS equivalent) to each server
  • 119. Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • 120. Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • 121. Simplify further Development Internet Stage server server (in the office) Front end Back end Perlbal Web server load balancer / (static+dynamic) proxy Database Cache server Back end Web server (static+dynamic)
  • 122. Waiters: cheaper, faster and less demanding
  • 123. Waiters: cheaper, faster and less demanding Internet Kids
  • 124. Waiters: cheaper, faster and less demanding Front end Internet Waiters - good at serving Kids quickly and dealing with annoying customers or Kitchens
  • 125. Waiters: cheaper, faster and less demanding Kitchen 1 Kitchen 2 Front end Internet Waiters - good at serving Kids Back end quickly and dealing with annoying customers or Chefs - simple and complex meals (static and dynamic) Kitchens
  • 126. Waiters: cheaper, faster and less demanding Kitchen 1 Kitchen 2 Front end Internet Waiters - good at serving Kids Back end quickly and dealing with Cache annoying customers or Chefs - simple and complex meals Freezer (static and dynamic) Kitchens
  • 127. More tips for faster user experience
  • 128. mod_gzip
  • 129. mod_gzip HTML/css/xml
  • 130. mod_gzip HTML/css/xml
  • 131. mod_gzip gzip HTML/css/xml
  • 132. mod_gzip gzip HTML/css/xml
  • 133. mod_gzip gzip HTML/css/xml Users
  • 134. mod_gzip gzip HTML/css/xml Users If a browser support compressions... compress (javascript can be tricky - at least minify)
  • 135. Cache headers (expiry)
  • 136. Cache headers (expiry)
  • 137. Cache headers (expiry) Let web browsers know how long to cache
  • 138. Cache somethings forever
  • 139. Cache somethings forever /includes/js/<VERSION>/common.js
  • 140. Cache somethings forever /includes/js/<VERSION>/common.js Ensures user has version which matches HTML (client could have cache of old HTML)
  • 141. Cache somethings forever /includes/js/<VERSION>/common.js Ensures user has version which matches HTML (client could have cache of old HTML) Use include file to update all pages
  • 142. Handling images: MogileFS
  • 143. Handling images: MogileFS Store images by group (replication levels)
  • 144. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers
  • 145. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server
  • 146. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user
  • 147. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user Automatic resilience
  • 148. Handling images: MogileFS Store images by group (replication levels) Spread IO across disks & servers Retrieve from fastest server Integrate with Perlbal, hidden from the user Automatic resilience
  • 149. Basic rules to consider
  • 150. Basic rules to consider Centralise (code/templates/configuration/ deployment etc)
  • 151. Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them
  • 152. Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great
  • 153. Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great Keep is simple (perlbal especially)
  • 154. Basic rules to consider Centralise (code/templates/configuration/ deployment etc) Test, run lots of them Cache if you need it; memcache is great Keep is simple (perlbal especially) Sometimes it's just easier to buy another server (this is not always true)
  • 155. We covered...
  • 156. We covered... perlbal
  • 157. We covered... perlbal memcached
  • 158. We covered... perlbal memcached svk (subversion)
  • 159. We covered... perlbal memcached svk (subversion) trac
  • 160. We covered... perlbal memcached svk (subversion) trac rsync
  • 161. We covered... perlbal memcached svk (subversion) trac rsync gzip
  • 162. We covered... perlbal mogileFS memcached svk (subversion) trac rsync gzip
  • 163. We covered... perlbal mogileFS server architecture memcached svk (subversion) trac rsync gzip
  • 164. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) trac rsync gzip
  • 165. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac rsync gzip
  • 166. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync gzip
  • 167. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync HTTP Cache headers gzip
  • 168. We covered... perlbal mogileFS server architecture memcached Test::More svk (subversion) mod_perl trac Template::Toolkit rsync HTTP Cache headers gzip and...
  • 169. ...how a kitchen should run
  • 170. ...how a kitchen should run
  • 171. ...how a kitchen should run Any questions?
  • 172. Evolving architecture http://leo.cuckoo.org/projects/ea/ Leo Lapworth http://leo.cuckoo.org/