The document discusses Gosuke Miyashita's goal of building a scalable storage system for his company's web hosting service. He is exploring the use of several open source technologies including cman, CLVM, GFS2, GNBD, DRBD, and DM-MP to create a storage system that provides high availability, flexible I/O distribution, and easy extensibility without expensive hardware. He outlines how each technology works and shows some example configurations, but notes that integrating many components may introduce issues around complexity, overhead, performance, stability and compatibility with non-Red Hat Linux.

1. TLUG Meeting 2008/09/13 Gosuke Miyashita

2. My company paperboy&co. Web hosting, blog, ec hosting and so on for indivisuals About 1,000 Linux servers Many single servers ...

3. My goal of a scalable storage system Storage system for a web hosting service High resource availability Flexible I/O distribution Easy to extend Mountable by multiple hosts No SPoF With OSS Without expensive hardwares Now I’m trying technologies for these purposes

5. cman CLVM GFS2 GNBD DRBD DM-MP Technologies

6. cman Cluster Manager A component of Red Hat Cluster Suit Membership management Messaging among cluster nodes Needed for CLVM and GFS2

7. CLVM Cluster Logical Volume Manager Cluster-wide version of LVM2 Automatically share LVM2 metadata among all cluster nodes So logical volumes with CLVM available to all cluster nodes

8. CLVM clvmd distributes metadata among cluster nodes Logical volumes presented to each cluster node Logical volume on shared storage LVM2 Metadata clvmd LVM2 Metadata clvmd LVM2 Metadata clvmd

9. GNBD Global Network Block Device Provides block-device access over TCP/IP Similar to iSCSI Advantage over iSCSI is built-in fencing

10. GNBD TCP/IP network GNBD client GNBD client GNBD client GNBD Server Exported block device

11. GFS2 Global File System 2 One of cluster-aware file systems Multiple nodes can simultaneously access this filesystem Uses DLM(Distributed Lock Manager) of cman to maintain file system integrity OCFS is another cluster-aware file system

12. GFS2 These nodes can access to the GFS2 file system simultaneously GNBD Server GFS2 GNBD client cman GNBD client cman GNBD client cman

13. DRBD Distributed Replicated Block Device RAID1 over a network Mirrors a whole block device over TCP/IP Available Active/Active with cluster file systems

14. DRBD Replication Server Block Device Server Block Device

15. DM-MP Device-Mapper Multipath Bundles I/O paths to one virtual I/O path Can choose active/passive or active/active

16. DM-MP with SAN storage /dev/sda1 /dev/sdb1 Seen as one device /dev/mapper/mpath0 active/passive or active/active Node HBA1 HBA2 SAN swtich 1 SAN swtich 2 Storage CNTRLR1 CNTRLR2

18. A scalable storage system /dev/VG0/LV0 (CLVM) mount /dev/VG0/LV0 /mnt cman GNBD cman GNBD GNBD Server GFS2 GNBD Server GFS2 Replication (DRBD) /dev/mapper/mpath0 (DM-MP) /dev/gnbd0 /dev/gnbd1 GNBD Server GFS2 GNBD Server GFS2 Replication (DRBD) /dev/mapper/mpath1 (DM-MP) /dev/gnbd2 /dev/gnbd3

19. How to extend /dev/VG0/LV0 (CLVM) mount /dev/VG0/LV0 /mnt cman GNBD cman GNBD GNBD Server GFS2 GNBD Server GFS2 /dev/mapper/mpath0 /dev/gnbd0 /dev/gnbd1 GNBD Server GFS2 GNBD Server GFS2 /dev/mapper/mpath1 /dev/gnbd2 /dev/gnbd3 GNBD Server GFS2 GNBD Server GFS2 /dev/mapper/mpath2 /dev/gnbd4 /dev/gnbd5

21. I wonder ... Many components cause troubles? How about overhead and performance? How about stability? More better way? How about other than Red Hat Linux?