Your SlideShare is downloading. ×
Changing Your Tires at 100mph Scaling Typepad
Or... Changing Your Tires at 161kph
Oversights <ul><li>Mars Climate Orbiter </li></ul><ul><li>Failure to convert English to Metric </li></ul><ul><li>Mighty cr...
Lessons <ul><li>Small details are important </li></ul><ul><li>Every mistake is a learning experience </li></ul><ul><li>Suc...
Typepad Enterprise Hosted Blogs
Technical Details <ul><li>Linux </li></ul><ul><li>Apache </li></ul><ul><li>Postgres </li></ul><ul><li>Perl </li></ul>= LAPP
Original Storage Setup 0 MySQL MySQL 1 2 User App Database Storage NFS NFS NFS Apache mod_perl Postgres
Single Server Solution App Database Storage NFS Apache mod_perl Postgres
Meanwhile…. <ul><li>Growing user base </li></ul><ul><li>Growing activity </li></ul><ul><li>Growing data </li></ul>
Mighty Crash <ul><li>Storage Failure </li></ul><ul><li>“ Split Brain” problem on redundant filers </li></ul><ul><li>RAID c...
Replicated and Partitioned <ul><li>Replicated MySQL Clusters </li></ul><ul><li>Partition data by user ID </li></ul><ul><li...
“ HA” Database Configuration <ul><li>MySQL 5.0 </li></ul><ul><li>InnoDB </li></ul><ul><li>Master-master replication </li><...
Redefinition App Database Storage NFS Apache mod_perl User General Global Postgres 0 MySQL
Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
The Tricky Part <ul><li>Need to move users without downtime </li></ul><ul><li>User 'read-only' mode </li></ul><ul><li>Chan...
Data Checks <ul><li>Row level data checking </li></ul><ul><li>Publish checks </li></ul>
Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
All Users Moved App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2
Additional 'Misc' Cluster App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2 My...
Postgres Eliminated App Database Storage NFS Apache mod_perl User General Global MySQL 0 MySQL MySQL 1 2 MySQL
Migration to Mogile <ul><li>Distributed </li></ul><ul><li>Redundant </li></ul><ul><li>Replicated  </li></ul><ul><li>Open s...
Blog Serving Blogs Storage NFS Apache2
Migration Process <ul><li>New images get saved to MogileFS </li></ul><ul><li>Backend process copies existing images to Mog...
Old Architecture Blogs Storage NFS Apache2
Addition of MogileFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored
Scenarios <ul><li>Serving from NFS </li></ul><ul><li>Serving from Mogile </li></ul>
Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 1 2
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 5 1 2
Serving From Mogile <ul><li>HTTP Headers </li></ul><ul><li>X-REPROXY-URL </li></ul><ul><li>X-REPROXY-CACHE-FOR </li></ul>B...
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 8 1
Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 9 8 1
More Memcached <ul><li>Data::ObjectDriver </li></ul><ul><li>Counts </li></ul><ul><li>Sets </li></ul><ul><li>Stats </li></ul>
Use the Schwartz <ul><li>Moblogging </li></ul><ul><li>Future publishing </li></ul><ul><li>Cache invalidation </li></ul><ul...
Gains <ul><li>Engineering </li></ul><ul><li>Easily turn complex tasks into Schwartz Events </li></ul><ul><li>Memcached hea...
Upcoming SlideShare
Loading in...5
×

How Typepad changed their architecture without taking down the service

3,345

Published on

Are you pushing the envelope of what your web application can handle? Do your engineers feel the impending need to overhaul and retrofit your system before it unexpectedly keels over on you? Come hear Six Apart talk about the tools, software and most importantly, the process it has been using to completely rewrite the backend of TypePad without disrupting thousands of users and paying customers.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,345
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "How Typepad changed their architecture without taking down the service"

  1. 1. Changing Your Tires at 100mph Scaling Typepad
  2. 2. Or... Changing Your Tires at 161kph
  3. 3. Oversights <ul><li>Mars Climate Orbiter </li></ul><ul><li>Failure to convert English to Metric </li></ul><ul><li>Mighty crash </li></ul>
  4. 4. Lessons <ul><li>Small details are important </li></ul><ul><li>Every mistake is a learning experience </li></ul><ul><li>Success requires coordination and cooperation </li></ul>
  5. 5. Typepad Enterprise Hosted Blogs
  6. 6. Technical Details <ul><li>Linux </li></ul><ul><li>Apache </li></ul><ul><li>Postgres </li></ul><ul><li>Perl </li></ul>= LAPP
  7. 7. Original Storage Setup 0 MySQL MySQL 1 2 User App Database Storage NFS NFS NFS Apache mod_perl Postgres
  8. 8. Single Server Solution App Database Storage NFS Apache mod_perl Postgres
  9. 9. Meanwhile…. <ul><li>Growing user base </li></ul><ul><li>Growing activity </li></ul><ul><li>Growing data </li></ul>
  10. 10. Mighty Crash <ul><li>Storage Failure </li></ul><ul><li>“ Split Brain” problem on redundant filers </li></ul><ul><li>RAID controller fails </li></ul><ul><li>Garbage strewn across all RAID disks </li></ul><ul><li>Database Failures </li></ul><ul><li>Data corruption </li></ul><ul><li>Corruption copied to backup </li></ul>
  11. 11. Replicated and Partitioned <ul><li>Replicated MySQL Clusters </li></ul><ul><li>Partition data by user ID </li></ul><ul><li>Global DB to map user to partition </li></ul><ul><li>Sequence generation by global </li></ul><ul><li>Map other data into 'roles' </li></ul>
  12. 12. “ HA” Database Configuration <ul><li>MySQL 5.0 </li></ul><ul><li>InnoDB </li></ul><ul><li>Master-master replication </li></ul><ul><li>Linux Heartbeat for shared vip/failover </li></ul><ul><li>Even/odd auto increment </li></ul>
  13. 13. Redefinition App Database Storage NFS Apache mod_perl User General Global Postgres 0 MySQL
  14. 14. Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
  15. 15. The Tricky Part <ul><li>Need to move users without downtime </li></ul><ul><li>User 'read-only' mode </li></ul><ul><li>Changes from the app saved in-line </li></ul><ul><li>Comments saved as schwartz jobs </li></ul>
  16. 16. Data Checks <ul><li>Row level data checking </li></ul><ul><li>Publish checks </li></ul>
  17. 17. Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
  18. 18. All Users Moved App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2
  19. 19. Additional 'Misc' Cluster App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2 MySQL
  20. 20. Postgres Eliminated App Database Storage NFS Apache mod_perl User General Global MySQL 0 MySQL MySQL 1 2 MySQL
  21. 21. Migration to Mogile <ul><li>Distributed </li></ul><ul><li>Redundant </li></ul><ul><li>Replicated </li></ul><ul><li>Open source </li></ul><ul><li>Commodity Hardware </li></ul><ul><li>For Typepad, images only </li></ul>
  22. 22. Blog Serving Blogs Storage NFS Apache2
  23. 23. Migration Process <ul><li>New images get saved to MogileFS </li></ul><ul><li>Backend process copies existing images to MogileFS </li></ul><ul><li>Must serve from both NFS and MogileFS </li></ul>
  24. 24. Old Architecture Blogs Storage NFS Apache2
  25. 25. Addition of MogileFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored
  26. 26. Scenarios <ul><li>Serving from NFS </li></ul><ul><li>Serving from Mogile </li></ul>
  27. 27. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
  28. 28. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
  29. 29. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
  30. 30. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 1
  31. 31. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
  32. 32. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
  33. 33. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
  34. 34. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 1 2
  35. 35. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 5 1 2
  36. 36. Serving From Mogile <ul><li>HTTP Headers </li></ul><ul><li>X-REPROXY-URL </li></ul><ul><li>X-REPROXY-CACHE-FOR </li></ul>Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 1
  37. 37. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 1
  38. 38. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 8 1
  39. 39. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 9 8 1
  40. 40. More Memcached <ul><li>Data::ObjectDriver </li></ul><ul><li>Counts </li></ul><ul><li>Sets </li></ul><ul><li>Stats </li></ul>
  41. 41. Use the Schwartz <ul><li>Moblogging </li></ul><ul><li>Future publishing </li></ul><ul><li>Cache invalidation </li></ul><ul><li>Publishing </li></ul>
  42. 42. Gains <ul><li>Engineering </li></ul><ul><li>Easily turn complex tasks into Schwartz Events </li></ul><ul><li>Memcached heavyweight data </li></ul><ul><li>Ops </li></ul><ul><li>Can easily add more machines </li></ul><ul><li>Can easily adjust workload </li></ul><ul><li>Fewer single points of failure, and cheaper </li></ul>

×