• Save
Os Webb
Upcoming SlideShare
Loading in...5
×
 

Os Webb

on

  • 1,605 views

 

Statistics

Views

Total Views
1,605
Views on SlideShare
1,605
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Os Webb Os Webb Presentation Transcript

  • Changing Your Tires at 100mph Scaling Typepad
  • Or... Changing Your Tires at 161kph
  • Oversights
    • Mars Climate Orbiter
    • Failure to convert English to Metric
    • Mighty crash
  • Lessons
    • Small details are important
    • Every mistake is a learning experience
    • Success requires coordination and cooperation
  • Typepad Enterprise Hosted Blogs
  • Technical Details
    • Linux
    • Apache
    • Postgres
    • Perl
    = LAPP
  • Original Storage Setup 0 MySQL MySQL 1 2 User App Database Storage NFS NFS NFS Apache mod_perl Postgres
  • Single Server Solution App Database Storage NFS Apache mod_perl Postgres
  • Meanwhile….
    • Growing user base
    • Growing activity
    • Growing data
  • Mighty Crash
    • Storage Failure
    • “ Split Brain” problem on redundant filers
    • RAID controller fails
    • Garbage strewn across all RAID disks
    • Database Failures
    • Data corruption
    • Corruption copied to backup
  • Replicated and Partitioned
    • Replicated MySQL Clusters
    • Partition data by user ID
    • Global DB to map user to partition
    • Sequence generation by global
    • Map other data into 'roles'
  • “ HA” Database Configuration
    • MySQL 5.0
    • InnoDB
    • Master-master replication
    • Linux Heartbeat for shared vip/failover
    • Even/odd auto increment
  • Redefinition App Database Storage NFS Apache mod_perl User General Global Postgres 0 MySQL
  • Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
  • The Tricky Part
    • Need to move users without downtime
    • User 'read-only' mode
    • Changes from the app saved in-line
    • Comments saved as schwartz jobs
  • Data Checks
    • Row level data checking
    • Publish checks
  • Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL
  • All Users Moved App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2
  • Additional 'Misc' Cluster App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2 MySQL
  • Postgres Eliminated App Database Storage NFS Apache mod_perl User General Global MySQL 0 MySQL MySQL 1 2 MySQL
  • Migration to Mogile
    • Distributed
    • Redundant
    • Replicated
    • Open source
    • Commodity Hardware
    • For Typepad, images only
  • Blog Serving Blogs Storage NFS Apache2
  • Migration Process
    • New images get saved to MogileFS
    • Backend process copies existing images to MogileFS
    • Must serve from both NFS and MogileFS
  • Old Architecture Blogs Storage NFS Apache2
  • Addition of MogileFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored
  • Scenarios
    • Serving from NFS
    • Serving from Mogile
  • Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
  • Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
  • Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
  • Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 1
  • Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1
  • Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1
  • Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1
  • Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 1 2
  • Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 5 1 2
  • Serving From Mogile
    • HTTP Headers
    • X-REPROXY-URL
    • X-REPROXY-CACHE-FOR
    Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 1
  • Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 1
  • Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 8 1
  • Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 9 8 1
  • More Memcached
    • Data::ObjectDriver
    • Counts
    • Sets
    • Stats
  • Use the Schwartz
    • Moblogging
    • Future publishing
    • Cache invalidation
    • Publishing
  • Gains
    • Engineering
    • Easily turn complex tasks into Schwartz Events
    • Memcached heavyweight data
    • Ops
    • Can easily add more machines
    • Can easily adjust workload
    • Fewer single points of failure, and cheaper