Your SlideShare is downloading. ×
Os Webb
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Os Webb

1,012
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,012
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Changing Your Tires at 100mph Scaling Typepad
  • 2. Or... Changing Your Tires at 161kph
  • 3. Oversights
    • Mars Climate Orbiter
    • Failure to convert English to Metric
    • Mighty crash
  • 4. Lessons
    • Small details are important
    • Every mistake is a learning experience
    • Success requires coordination and cooperation
  • 5. Typepad Enterprise Hosted Blogs
  • 6. Technical Details
    • Linux
    • Apache
    • Postgres
    • Perl
    = LAPP
  • 7. Original Storage Setup 0 MySQL MySQL 1 2 User App Database Storage NFS NFS NFS Apache mod_perl Postgres
  • 8. Single Server Solution App Database Storage NFS Apache mod_perl Postgres
  • 9. Meanwhile….
    • Growing user base
    • Growing activity
    • Growing data
  • 10. Mighty Crash
    • Storage Failure
    • “ Split Brain” problem on redundant filers
    • RAID controller fails
    • Garbage strewn across all RAID disks
    • Database Failures
    • Data corruption
    • Corruption copied to backup
  • 11. Replicated and Partitioned
    • Replicated MySQL Clusters
    • Partition data by user ID
    • Global DB to map user to partition
    • Sequence generation by global
    • Map other data into 'roles'
  • 12. “ HA” Database Configuration
    • MySQL 5.0
    • InnoDB
    • Master-master replication
    • Linux Heartbeat for shared vip/failover
    • Even/odd auto increment
  • 13. Redefinition App Database Storage NFS Apache mod_perl User General Global Postgres 0 MySQL
  • 14. Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
  • 15. The Tricky Part
    • Need to move users without downtime
    • User 'read-only' mode
    • Changes from the app saved in-line
    • Comments saved as schwartz jobs
  • 16. Data Checks
    • Row level data checking
    • Publish checks
  • 17. Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
  • 18. All Users Moved App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2
  • 19. Additional 'Misc' Cluster App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2 MySQL
  • 20. Postgres Eliminated App Database Storage NFS Apache mod_perl User General Global MySQL 0 MySQL MySQL 1 2 MySQL
  • 21. Migration to Mogile
    • Distributed
    • Redundant
    • Replicated
    • Open source
    • Commodity Hardware
    • For Typepad, images only
  • 22. Blog Serving Blogs Storage NFS Apache2
  • 23. Migration Process
    • New images get saved to MogileFS
    • Backend process copies existing images to MogileFS
    • Must serve from both NFS and MogileFS
  • 24. Old Architecture Blogs Storage NFS Apache2
  • 25. Addition of MogileFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored
  • 26. Scenarios
    • Serving from NFS
    • Serving from Mogile
  • 27. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
  • 28. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
  • 29. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
  • 30. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 1
  • 31. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
  • 32. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
  • 33. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
  • 34. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 1 2
  • 35. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 5 1 2
  • 36. Serving From Mogile
    • HTTP Headers
    • X-REPROXY-URL
    • X-REPROXY-CACHE-FOR
    Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 1
  • 37. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 1
  • 38. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 8 1
  • 39. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 9 8 1
  • 40. More Memcached
    • Data::ObjectDriver
    • Counts
    • Sets
    • Stats
  • 41. Use the Schwartz
    • Moblogging
    • Future publishing
    • Cache invalidation
    • Publishing
  • 42. Gains
    • Engineering
    • Easily turn complex tasks into Schwartz Events
    • Memcached heavyweight data
    • Ops
    • Can easily add more machines
    • Can easily adjust workload
    • Fewer single points of failure, and cheaper