Typepad migrated their blog hosting infrastructure from a single server solution to a distributed, redundant, and replicated architecture using MySQL, Postgres, MogileFS, and Memcached to address growth and reliability issues. They moved user data and images to these distributed systems without downtime through a multi-step process. This allowed their infrastructure to easily scale and improved reliability by eliminating single points of failure.

1. Changing Your Tires at 100mph Scaling Typepad

2. Or... Changing Your Tires at 161kph

3. Oversights Mars Climate Orbiter Failure to convert English to Metric Mighty crash

4. Lessons Small details are important Every mistake is a learning experience Success requires coordination and cooperation

5. Typepad Enterprise Hosted Blogs

6. Technical Details Linux Apache Postgres Perl = LAPP

7. Original Storage Setup 0 MySQL MySQL 1 2 User App Database Storage NFS NFS NFS Apache mod_perl Postgres

8. Single Server Solution App Database Storage NFS Apache mod_perl Postgres

9. Meanwhile…. Growing user base Growing activity Growing data

10. Mighty Crash Storage Failure “ Split Brain” problem on redundant filers RAID controller fails Garbage strewn across all RAID disks Database Failures Data corruption Corruption copied to backup

11. Replicated and Partitioned Replicated MySQL Clusters Partition data by user ID Global DB to map user to partition Sequence generation by global Map other data into 'roles'

12. “ HA” Database Configuration MySQL 5.0 InnoDB Master-master replication Linux Heartbeat for shared vip/failover Even/odd auto increment

13. Redefinition App Database Storage NFS Apache mod_perl User General Global Postgres 0 MySQL

14. Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL

15. The Tricky Part Need to move users without downtime User 'read-only' mode Changes from the app saved in-line Comments saved as schwartz jobs

16. Data Checks Row level data checking Publish checks

17. Additional User Clusters App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL 1 2 MySQL

18. All Users Moved App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2

19. Additional 'Misc' Cluster App Database Storage NFS Apache mod_perl User General Global MySQL Postgres 0 MySQL MySQL 1 2 MySQL

20. Postgres Eliminated App Database Storage NFS Apache mod_perl User General Global MySQL 0 MySQL MySQL 1 2 MySQL

21. Migration to Mogile Distributed Redundant Replicated Open source Commodity Hardware For Typepad, images only

22. Blog Serving Blogs Storage NFS Apache2

23. Migration Process New images get saved to MogileFS Backend process copies existing images to MogileFS Must serve from both NFS and MogileFS

24. Old Architecture Blogs Storage NFS Apache2

25. Addition of MogileFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored

26. Scenarios Serving from NFS Serving from Mogile

27. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1

28. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1

29. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1

30. Serving From NFS Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 1

31. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 1

32. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 1

33. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 1

34. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 1 2

35. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 3 4 5 1 2

36. Serving From Mogile HTTP Headers X-REPROXY-URL X-REPROXY-CACHE-FOR Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 1

37. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 1

38. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 8 1

39. Serving From Mogile Blogs Storage NFS Apache2 mod_perl2 Mogile DB Perlbal Mogstored 2 3 4 5 6 7 9 8 1

40. More Memcached Data::ObjectDriver Counts Sets Stats

41. Use the Schwartz Moblogging Future publishing Cache invalidation Publishing

42. Gains Engineering Easily turn complex tasks into Schwartz Events Memcached heavyweight data Ops Can easily add more machines Can easily adjust workload Fewer single points of failure, and cheaper