Changing Your Tires at 100mph Scaling Typepad
Or... Changing Your Tires at 161kph
Oversights <ul><li>Mars Climate Orbiter </li></ul><ul><li>Failure to convert English to Metric </li></ul><ul><li>Mighty cr...
Lessons <ul><li>Small details are important </li></ul><ul><li>Every mistake is a learning experience </li></ul><ul><li>Suc...
Typepad Enterprise Hosted Blogs
Technical Details <ul><li>Linux </li></ul><ul><li>Apache </li></ul><ul><li>Postgres </li></ul><ul><li>Perl </li></ul>= LAPP
Original Storage Setup 0 MySQL MySQL 1 2 User App Database Storage NFS NFS NFS Apache mod_perl Postgres
Single Server Solution App Database Storage NFS Apache mod_perl Postgres
Meanwhile…. <ul><li>Growing user base </li></ul><ul><li>Growing activity </li></ul><ul><li>Growing data </li></ul>
Mighty Crash <ul><li>Storage Failure </li></ul><ul><li>“ Split Brain” problem on redundant filers </li></ul><ul><li>RAID c...
Replicated and Partitioned <ul><li>Replicated MySQL Clusters </li></ul><ul><li>Partition data by user ID </li></ul><ul><li...
“ HA” Database Configuration <ul><li>MySQL 5.0 </li></ul><ul><li>InnoDB </li></ul><ul><li>Master-master replication </li><...
Redefinition App Database Storage NFS Apache mod_perl User General Global Postgres 0 MySQL
Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
The Tricky Part <ul><li>Need to move users without downtime </li></ul><ul><li>User 'read-only' mode </li></ul><ul><li>Chan...
Data Checks <ul><li>Row level data checking </li></ul><ul><li>Publish checks </li></ul>
Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
All Users Moved App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2
Additional 'Misc' Cluster App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2 My...
Postgres Eliminated App Database Storage NFS Apache mod_perl User General Global MySQL 0 MySQL MySQL 1 2 MySQL
Migration to Mogile <ul><li>Distributed </li></ul><ul><li>Redundant </li></ul><ul><li>Replicated  </li></ul><ul><li>Open s...
Blog Serving Blogs Storage NFS Apache2
Migration Process <ul><li>New images get saved to MogileFS </li></ul><ul><li>Backend process copies existing images to Mog...
Old Architecture Blogs Storage NFS Apache2
Addition of MogileFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored
Scenarios <ul><li>Serving from NFS </li></ul><ul><li>Serving from Mogile </li></ul>
Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 1 2
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 5 1 2
Serving From Mogile <ul><li>HTTP Headers </li></ul><ul><li>X-REPROXY-URL </li></ul><ul><li>X-REPROXY-CACHE-FOR </li></ul>B...
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 8 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 9 8 1
More Memcached <ul><li>Data::ObjectDriver </li></ul><ul><li>Counts </li></ul><ul><li>Sets </li></ul><ul><li>Stats </li></ul>
Use the Schwartz <ul><li>Moblogging </li></ul><ul><li>Future publishing </li></ul><ul><li>Cache invalidation </li></ul><ul...
Gains <ul><li>Engineering </li></ul><ul><li>Easily turn complex tasks into Schwartz Events </li></ul><ul><li>Memcached hea...
Upcoming SlideShare
Loading in...5
×

Os Webb

1,031

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,031
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Os Webb"

  1. 1. Changing Your Tires at 100mph Scaling Typepad
  2. 2. Or... Changing Your Tires at 161kph
  3. 3. Oversights <ul><li>Mars Climate Orbiter </li></ul><ul><li>Failure to convert English to Metric </li></ul><ul><li>Mighty crash </li></ul>
  4. 4. Lessons <ul><li>Small details are important </li></ul><ul><li>Every mistake is a learning experience </li></ul><ul><li>Success requires coordination and cooperation </li></ul>
  5. 5. Typepad Enterprise Hosted Blogs
  6. 6. Technical Details <ul><li>Linux </li></ul><ul><li>Apache </li></ul><ul><li>Postgres </li></ul><ul><li>Perl </li></ul>= LAPP
  7. 7. Original Storage Setup 0 MySQL MySQL 1 2 User App Database Storage NFS NFS NFS Apache mod_perl Postgres
  8. 8. Single Server Solution App Database Storage NFS Apache mod_perl Postgres
  9. 9. Meanwhile…. <ul><li>Growing user base </li></ul><ul><li>Growing activity </li></ul><ul><li>Growing data </li></ul>
  10. 10. Mighty Crash <ul><li>Storage Failure </li></ul><ul><li>“ Split Brain” problem on redundant filers </li></ul><ul><li>RAID controller fails </li></ul><ul><li>Garbage strewn across all RAID disks </li></ul><ul><li>Database Failures </li></ul><ul><li>Data corruption </li></ul><ul><li>Corruption copied to backup </li></ul>
  11. 11. Replicated and Partitioned <ul><li>Replicated MySQL Clusters </li></ul><ul><li>Partition data by user ID </li></ul><ul><li>Global DB to map user to partition </li></ul><ul><li>Sequence generation by global </li></ul><ul><li>Map other data into 'roles' </li></ul>
  12. 12. “ HA” Database Configuration <ul><li>MySQL 5.0 </li></ul><ul><li>InnoDB </li></ul><ul><li>Master-master replication </li></ul><ul><li>Linux Heartbeat for shared vip/failover </li></ul><ul><li>Even/odd auto increment </li></ul>
  13. 13. Redefinition App Database Storage NFS Apache mod_perl User General Global Postgres 0 MySQL
  14. 14. Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
  15. 15. The Tricky Part <ul><li>Need to move users without downtime </li></ul><ul><li>User 'read-only' mode </li></ul><ul><li>Changes from the app saved in-line </li></ul><ul><li>Comments saved as schwartz jobs </li></ul>
  16. 16. Data Checks <ul><li>Row level data checking </li></ul><ul><li>Publish checks </li></ul>
  17. 17. Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
  18. 18. All Users Moved App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2
  19. 19. Additional 'Misc' Cluster App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2 MySQL
  20. 20. Postgres Eliminated App Database Storage NFS Apache mod_perl User General Global MySQL 0 MySQL MySQL 1 2 MySQL
  21. 21. Migration to Mogile <ul><li>Distributed </li></ul><ul><li>Redundant </li></ul><ul><li>Replicated </li></ul><ul><li>Open source </li></ul><ul><li>Commodity Hardware </li></ul><ul><li>For Typepad, images only </li></ul>
  22. 22. Blog Serving Blogs Storage NFS Apache2
  23. 23. Migration Process <ul><li>New images get saved to MogileFS </li></ul><ul><li>Backend process copies existing images to MogileFS </li></ul><ul><li>Must serve from both NFS and MogileFS </li></ul>
  24. 24. Old Architecture Blogs Storage NFS Apache2
  25. 25. Addition of MogileFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored
  26. 26. Scenarios <ul><li>Serving from NFS </li></ul><ul><li>Serving from Mogile </li></ul>
  27. 27. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
  28. 28. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
  29. 29. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
  30. 30. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 1
  31. 31. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
  32. 32. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
  33. 33. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
  34. 34. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 1 2
  35. 35. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 5 1 2
  36. 36. Serving From Mogile <ul><li>HTTP Headers </li></ul><ul><li>X-REPROXY-URL </li></ul><ul><li>X-REPROXY-CACHE-FOR </li></ul>Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 1
  37. 37. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 1
  38. 38. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 8 1
  39. 39. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 9 8 1
  40. 40. More Memcached <ul><li>Data::ObjectDriver </li></ul><ul><li>Counts </li></ul><ul><li>Sets </li></ul><ul><li>Stats </li></ul>
  41. 41. Use the Schwartz <ul><li>Moblogging </li></ul><ul><li>Future publishing </li></ul><ul><li>Cache invalidation </li></ul><ul><li>Publishing </li></ul>
  42. 42. Gains <ul><li>Engineering </li></ul><ul><li>Easily turn complex tasks into Schwartz Events </li></ul><ul><li>Memcached heavyweight data </li></ul><ul><li>Ops </li></ul><ul><li>Can easily add more machines </li></ul><ul><li>Can easily adjust workload </li></ul><ul><li>Fewer single points of failure, and cheaper </li></ul>

×