Migrating from PostgreSQL to MySQL at Cocolog

908 views

Published on

NIFTY Corporation will discuss their experiences with MySQL and how they changed the configuration in TypePad to meet the Japanese market’s requirement, such as more availability, more cell phone support, and so on.

Migration from PostgreSQL to MySQL, administration, optimization, and maintenance are also covered.

Through this session, the audience will recognize how to deal with scenarios when the number of servers grows quickly from six to three hundred, and also learn about real-life security and administration best practices.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Migrating from PostgreSQL to MySQL at Cocolog

  1. 1. Migrating from PostgreSQL to MySQLat CocologNaoto Yokoyama, NIFTY CorporationGarth Webb, Six ApartLisa Phillips, Six ApartCredits:Kenji Hirohama, Sumisho Computer Systems Corp.
  2. 2. Agenda 1. What is Cocolog 2. History of Cocolog 3. DBP: Database Partitioning 4. Migration From PostgreSQL to MySQL
  3. 3. 1. What is Cocolog
  4. 4. What is Cocolog NIFTY Corporation Established in 1986 A Fujitsu Group Company NIFTY-Serve (licensed and interconnected with CompuServe) One of the largest ISPs in Japan Cocolog First blog community at a Japanese ISP Based on TypePad technology by SixApart Several hundred million PV/month History Dec/02/2003: Cocolog for ISP users launch Nov/24/2005: Cocolog Free for free launch April/05/2007: Cocolog for Mobile Phone launch
  5. 5. 2008/04700 Thousand UsersCocolog (Screenshot of home page)
  6. 6. Cocolog (Screenshot of home page)TypePadCocolog
  7. 7. Cocolog template sets
  8. 8. Cocolog Growth (User) ■Cocolog ■Cocolog Freephase1 phase2 phase3 phase4
  9. 9. Cocolog Growth (Entry) ■Cocolog ■Cocolog Freephase1 phase2 phase3 phase4
  10. 10. Technology at Cocolog Core System Linux 2.4/2.6 Apache 1.3/2.0/2.2 & mod_perl Perl 5.8+CPAN PostgreSQL 8.1 MySQL 5.0 memcached/TheSchwartz/cfengine Eco System LAMP,LAPP,Ruby+ActiveRecord, Capistrano Etc...
  11. 11. Monitoring Management Tool Proprietary in-house development with PostgreSQL, PHP,and Perl Monitoring points (order of priority) response time of each post number of spam comments/trackbacks number of comments/trackbacks source IP address of spam number of entries number of comments via mobile devices page views via mobile devices time of batch completion amount of API usage bandwidth usage DB Disk I/O Memory and CPU usage time of VACUUM analyze APP number of active processes CPU usage Memory usageHardDBServiceAPL
  12. 12. 2. History of Cocolog
  13. 13. Phase1 2003/12~(Entry: 0.04Million)RegisterPostgreSQLNASWEBStatic contentsPublishedBefore DBP10serversTypePad
  14. 14. PodcastPortalProfileEtc..Phase2 2004/12~ (Entry: 7Million)Rich templatePublish BookTel OperatorSupportNASWEBStatic contentsPublishedPostgreSQLRegisterTypePad2004/12~2005/5~Before DBP50servers
  15. 15. Phase2 - Problems The system is tightly coupled. Database server is receiving from multiplepoints. It is difficult to change the system design anddatabase schema.
  16. 16. Phase3 2006/3~ (Entry: 12Million)NASWEBStatic contentsPublishedWeb-APImemcachedPodcastPortalProfileEtc..PostgreSQLRich templatePublish BookTel OperatorSupportRegisterTypePadBefore DBP200servers
  17. 17. Phase4 2007/4~ (Entry: 16Million)Web-APIStatic contentsPublishedmemcachedAtomMobileWEBRich templatePublish BookTel OperatorSupportRegisterTypepadPostgreSQLBefore DBP300servers
  18. 18. Now 2008/4~Web-APIStatic contentsPublishedmemcachedAtomMobileWEBTypepadRich templatePublish BookTel OperatorSupportRegisterMultiMySQLAfter DBP150servers
  19. 19. 3. TypePad Database Partitioning
  20. 20. Steps for Transitioning• Server PreparationHardware and software setup• Global WriteWrite user information to the global DB• Global ReadRead/write user information on the global DB• Move SequenceTable sequences served by global DB• User Data MoveMove user data to user partitions• New User PartitionAll new users saved directly to user partition 1• New User StrategyDecide on a strategy for the new user partition• Non User Data MoveMove all non-user owned data
  21. 21. StorageTypePad Overview (PreDBP)Database(Postgres)Static Content (HTML,Images, etc)ApplicationServerWebServerTypeCastServerATOMServerMEMCACHEDData Caching servers toreduce DB loadDedicated Server forTypeCast (via ATOM)https(443)http(80)http(80) : atom apimemcached(11211)postgres(5432)MailServerInternetnfs(2049)ADMIN(CRON)Serversmtp(25) / pop(110)Blog ReadersBlog OwnersMobile BlogReaderssmtp(25) / pop(110)Cron Server for periodicasynchronous tasks
  22. 22. TypePadTypePadTypePadNon-User RoleWhy Partition?TypePadUser Role(User0)All inquires (access) go to oneDB(Postgres)After DBPCurrent setupInquiries (access) are divided amongseveral DB(MySQL)TypePadTypePadTypePadTypePadGlobalRoleNon-UserRoleUser Role(User1)User Role(User2)User Role(User3)
  23. 23. Non-User RoleServer PreparationTypePadUser Role(User0)DB(PostgreSQL)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleNew expanded setupDB(MySQL) for partitioned dataCurrent SetupJob Server+ TypePad+ SchwartzSchwartzDBUser information ispartitionedMaintains user mappingand primary key generation Stores jobdetailsServer forexecuting Jobs※Grey areas are not used in currentstepsAsynchronous Job ServerInformation that does notneed to be partitioned(such as sessioninformation)
  24. 24. Global WriteCreating the user mapNon-User RoleTypePadUser Role(User0)DB(PostgreSQL)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleJob Server+ TypePad+ SchwartzSchwartzDB①②Explanation①:For new registrations only, uniquely identifying user data is written to the global DB②:This same data continues to be written to the existing DBDB(MySQL) for partitioned dataAsynchronous Job ServerMaintains user mappingand primary key generation※Grey areas are not used in current steps
  25. 25. Global ReadUse the user map to find the user partitionNon-User RoleTypePadUser Role(User0)DB(PostgreSQL)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleJob Server+ TypePad+ SchwartzSchwartzDBExplanation①:Migrate existing user data to the global DB②:At start of the request, the application queries global DB for the location of user data③:The application then talks to this DB for all queries about this user. At this stage the global DB pointsto the user0 partition in all cases.DB(MySQL) for partitioned dataMaintains user mappingand primary key generation①Migrate existinguser dataAsynchronous Job Server②③※Grey areas are not used in current steps
  26. 26. Move SequenceMigrating primary key generationNon-User RoleTypePadUser Role(User0)DB(PostgreSQL)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleJob Server+ TypePad+ SchwartzSchwartzDBExplanation①:Postgres sequences (for generating unique primary keys) are migrated to tables on the global DB thatact as “pseudo-sequences”.② Application requests new primary keys from global DB rather than the user partition.DB(MySQL) for partitioned dataMaintains user mappingand primary key generation①※Grey areas are not used in current stepsMigrate sequencemanagementAsynchronous Job Server②
  27. 27. User Data MoveMoving user data to the new user-role partitionsNon-User RoleTypePadUser Role(User0)DB(PostgreSQL)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleJob Server+ TypePad+ SchwartzSchwartzDBExplanation①:Existing users that should be migrated by Job Server are submitted as new Schwartz jobs. User data isthen migrated asynchronously②:If a comment arrives while the user is being migrated, it is saved in the Schwartz DB to be published later.③:After being migrated all user data will exist on the user-role DB partitions④:Once all user data is migrated, only non-user data is on PostgresDB(MySQL) for partitioned dataStores jobdetailsServer forexecuting JobsMaintains user mappingand primary key generationUser information ispartitioned①②※Grey areas are not used in current steps③Migrating eachuser dataDB(MySQL) for partitioned data④
  28. 28. New User PartitionNew registrations are created on one user role partitionNon-User RoleTypePadUser Role(User0)DB(PostgreSQL)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleJob Server+ TypePad+ SchwartzSchwartzDBExplanation①:When new users register, user data is written to a user role partition.②:Non-user data continues to be served off PostgresDB(MySQL) for partitioned dataMaintains user mappingand primary key generationUser information ispartitioned①②※Grey areas are not used in current stepsAsynchronous Job Server
  29. 29. New User StrategyPick a scheme for distributing new usersNon-User RoleTypePadUser Role(User0)DB(PostgreSQL)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleJob Server+ TypePad+ SchwartzSchwartzDBExplanation①:When new users register, user data is written to one of the user role partitions, depending on a setdistribution method (round robin, random, etc)②:Non-user data continues to be served off PostgresDB(MySQL) for partitioned dataMaintains user mappingand primary key generationUser information ispartitioned①②※Grey areas are not used in current stepsAsynchronous Job Server
  30. 30. Non User Data MoveMigrate data that cannot be partitioned by userNon-User RoleTypePadUser Role(User0)DB(PostgreSQL)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleJob Server+ TypePad+ SchwartzSchwartzDBExplanation①:Migrate non-user role data left on PostgreSQL to the MySQL side.DB(MySQL) for partitioned dataMaintains user mappingand primary key generationUser information ispartitioned①※Grey areas are not used in current stepsMigrate non-UserdataAsynchronous Job ServerInformation that does notneed to be partitioned(such as sessioninformation)
  31. 31. Data migration doneNon-User RoleTypePadUser Role(User0)DB(Postgres)User Role(User1)User Role(User2)User Role(User3)GlobalRoleNon-UserRoleJob Server+ TypePad+ SchwartzSchwartzDBExplanation①:All data access is now done through MySQL②:Continue to use The Schwartz for asynchronous jobsDB(MySQL) for partitioned dataStores jobdetailsServer forexecuting JobsMaintains user mappingand primary key generationUser information ispartitioned①※Grey areas are not used in current steps①② Asynchronous Job ServerInformation that does notneed to be partitioned(such as sessioninformation)
  32. 32. StorageThe New TypePad configurationDatabase(MySQL)Static Content(HTML,Images, etc)ApplicationServerWebServerTypeCastServerATOMServerMEMCACHEDData Caching servers toreduce DB loadDedicated Server forTypeCast (via ATOM)https(443)http(80)http(80) : atom apimemcached(11211)MySQL(3306)MailServerInternetnfs(2049)ADMIN(CRON)Serversmtp(25) / pop(110)Blog ReadersBlog Owners(managementinterface)Mobile BlogReaderssmtp(25) / pop(110)Cron Server for periodicasynchronous tasksJobServerTheSchwartz server forrunning ad-hoc jobsasynchronously
  33. 33. 4. Migration from PostgreSQL to MySQL
  34. 34.  DB Node Spec HistoryTime OS(RedHat) CPU Xeon MEM DiskArray2003/122007/117.4(2.4.9) 1.8GHz/512k×1 1GB NoES2.1(2.4.9) 3.2GHz/1M×2 4GB NoES2.1(2.4.9) 3.2GHz/1M×2 4GB YesAS2.1(2.4.9) 3.2GHz/1M×4 12GBYesAS4 (2.6.9) 3.2GHz/1M×4 12GBYesAS4 (2.6.9) MP3.3GHz/1M×4〔2Core×4〕16GBYesHistory of scale up PostgreSQL server, Before DBP
  35. 35.  DB DiskArray Spec [FUJITSU ETERNUS8000] Best I/O transaction performance in the world 146GB (15 krpm) * 32disk with RAID - 10 MultiPath FibreChannel 4Gbps QuickOPC (One Point Copy) OPC copy functions let you create a duplicate copyof any data from the original at any chosen time.http://www.computers.us.fujitsu.com/www/products_storage.shtml?products/storage/fujitsu/e8000/e8000History of scale up PostgreSQL server, Before DBP
  36. 36. Scale out MySQL servers, After DBP A role configuration Each role is configured as HA cluster HA Software: NEC ClusterPro Shared Storage
  37. 37. Scale out MySQL servers, After DBPPostgreSQLFibreChannel SANDiskArray…heart beatMySQLRole3MySQLRole2MySQLRole1TypePadApplication
  38. 38. Scale out MySQL servers, After DBP Backup Replication w/ Hot backup
  39. 39. Scale out MySQL servers, After DBPPostgreSQLFibreChannel SANDiskArray…heart beatMySQLRole3MySQLRole2MySQLRole1MySQLBackupRoleTypePadApplicationmysqld mysqld mysqldrep rep repopcmysqldmysqldmysqld
  40. 40. Troubles with PostreSQL 7.4 – 8.1 Data size over 100 GB 40% is index Severe Data Fragmentation VACUUM “VACUUM analyze” cause the performance problem Takes too long to VACUUM large amounts of data dump/restore is the only solution for de-fragmentation Auto VACUUM We don’t use Auto VACUUM since we are worried aboutlatent response time
  41. 41. Troubles with PostgreSQL 7.4 – 8.1 Character set PostgreSQL allow the out of boundary UTF-8Japanese extended character sets and multibytes character sets which normally shouldcome back with an error - instead ofaccepting them.
  42. 42. “Cleaning” data Removing characters set that are out of theboundries UTF-8 character sets. Steps PostgreSQL.dumpALL Split for Piconv UTF8 -> UCS2 -> UTF8 & Merge PostgreSQL.restoredump Split UTF8->UCS2->UTF8 Mergerestore
  43. 43. TypePadTypePadMigration from PostgreSQL to MySQL using TypePad script Steps PostgreSQL -> PerlObject & tmp publish-> MySQL -> PerlObject & last publish diff tmp & last Object (data check) diff tmp & last publish (file check)PostgreSQLDocumentObjecttmpDocumentObjectlastFile checkdata check
  44. 44. Troubles with MySQL convert_tz function doesnt support the input value outside thescope of Unix Time sort order different sort order without “order by” clause
  45. 45. Cocolog Future Plans Dynamic Job queue
  46. 46. Consulting by Sumisho Computer Systems Corp. System Integrator first and best partner of MySQL in Japansince 2003 provide MySQL consulting, support, trainingservice HA Maintenance online backup Japanese character support
  47. 47. Questions

×