Changing Your Tires at 100mph Scaling Typepad

Or... Changing Your Tires at 161kph

Oversights Mars Climate Orbiter Failure to convert English to Metric Mighty crash

Mars Climate Orbiter

Failure to convert English to Metric

Mighty crash

Lessons Small details are important Every mistake is a learning experience Success requires coordination and cooperation

Small details are important

Every mistake is a learning experience

Success requires coordination and cooperation

Typepad Enterprise Hosted Blogs

Technical Details Linux Apache Postgres Perl = LAPP

Linux

Apache

Postgres

Perl

Original Storage Setup 0 MySQL MySQL 1 2 User App Database Storage NFS NFS NFS Apache mod_perl Postgres

Single Server Solution App Database Storage NFS Apache mod_perl Postgres

Meanwhile…. Growing user base Growing activity Growing data

Growing user base

Growing activity

Growing data

Mighty Crash Storage Failure “ Split Brain” problem on redundant filers RAID controller fails Garbage strewn across all RAID disks Database Failures Data corruption Corruption copied to backup

Storage Failure

“ Split Brain” problem on redundant filers

RAID controller fails

Garbage strewn across all RAID disks

Database Failures

Data corruption

Corruption copied to backup

Replicated and Partitioned Replicated MySQL Clusters Partition data by user ID Global DB to map user to partition Sequence generation by global Map other data into 'roles'

Replicated MySQL Clusters

Partition data by user ID

Global DB to map user to partition

Sequence generation by global

Map other data into 'roles'

“ HA” Database Configuration MySQL 5.0 InnoDB Master-master replication Linux Heartbeat for shared vip/failover Even/odd auto increment

MySQL 5.0

InnoDB

Master-master replication

Linux Heartbeat for shared vip/failover

Even/odd auto increment

Redefinition App Database Storage NFS Apache mod_perl User General Global Postgres 0 MySQL

Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL

The Tricky Part Need to move users without downtime User 'read-only' mode Changes from the app saved in-line Comments saved as schwartz jobs

Need to move users without downtime

User 'read-only' mode

Changes from the app saved in-line

Comments saved as schwartz jobs

Data Checks Row level data checking Publish checks

Row level data checking

Publish checks

Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL

All Users Moved App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2

Additional 'Misc' Cluster App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2 MySQL

Postgres Eliminated App Database Storage NFS Apache mod_perl User General Global MySQL 0 MySQL MySQL 1 2 MySQL

Migration to Mogile Distributed Redundant Replicated Open source Commodity Hardware For Typepad, images only

Distributed

Redundant

Replicated

Open source

Commodity Hardware

For Typepad, images only

Blog Serving Blogs Storage NFS Apache2

Migration Process New images get saved to MogileFS Backend process copies existing images to MogileFS Must serve from both NFS and MogileFS

New images get saved to MogileFS

Backend process copies existing images to MogileFS

Must serve from both NFS and MogileFS

Old Architecture Blogs Storage NFS Apache2

Addition of MogileFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored

Scenarios Serving from NFS Serving from Mogile

Serving from NFS

Serving from Mogile

Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1

Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1

Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1

Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 1

Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1

Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1

Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1

Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 1 2

Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 5 1 2

Serving From Mogile HTTP Headers X-REPROXY-URL X-REPROXY-CACHE-FOR Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 1

HTTP Headers

X-REPROXY-URL

X-REPROXY-CACHE-FOR

Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 1

Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 8 1

Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 9 8 1

More Memcached Data::ObjectDriver Counts Sets Stats

Data::ObjectDriver

Counts

Sets

Stats

Use the Schwartz Moblogging Future publishing Cache invalidation Publishing

Moblogging

Future publishing

Cache invalidation

Publishing

Gains Engineering Easily turn complex tasks into Schwartz Events Memcached heavyweight data Ops Can easily add more machines Can easily adjust workload Fewer single points of failure, and cheaper

Engineering

Easily turn complex tasks into Schwartz Events

Memcached heavyweight data

Ops

Can easily add more machines

Can easily adjust workload

Fewer single points of failure, and cheaper