TLUG Meeting 2008/09/13 Gosuke Miyashita

My company paperboy&co. Web hosting, blog, ec hosting and so on for indivisuals About 1,000 Linux servers Many single servers ...

paperboy&co.

Web hosting, blog, ec hosting and so on for indivisuals

About 1,000 Linux servers

Many single servers ...

My goal of a scalable storage system Storage system for a web hosting service High resource availability Flexible I/O distribution Easy to extend Mountable by multiple hosts No SPoF With OSS Without expensive hardwares Now I’m trying technologies for these purposes

Storage system for a web hosting service

High resource availability

Flexible I/O distribution

Easy to extend

Mountable by multiple hosts

No SPoF

With OSS

Without expensive hardwares

Now I’m trying technologies for these purposes

cman CLVM GFS2 GNBD DRBD DM-MP Technologies

cman

CLVM

GFS2

GNBD

DRBD

DM-MP

cman Cluster Manager A component of Red Hat Cluster Suit Membership management Messaging among cluster nodes Needed for CLVM and GFS2

Cluster Manager

A component of Red Hat Cluster Suit

Membership management

Messaging among cluster nodes

Needed for CLVM and GFS2

CLVM Cluster Logical Volume Manager Cluster-wide version of LVM2 Automatically share LVM2 metadata among all cluster nodes So logical volumes with CLVM available to all cluster nodes

Cluster Logical Volume Manager

Cluster-wide version of LVM2

Automatically share LVM2 metadata among all cluster nodes

So logical volumes with CLVM available to all cluster nodes

CLVM clvmd distributes metadata among cluster nodes Logical volumes presented to each cluster node Logical volume on shared storage LVM2 Metadata clvmd LVM2 Metadata clvmd LVM2 Metadata clvmd

GNBD Global Network Block Device Provides block-device access over TCP/IP Similar to iSCSI Advantage over iSCSI is built-in fencing

Global Network Block Device

Provides block-device access over TCP/IP

Similar to iSCSI

Advantage over iSCSI is built-in fencing

GNBD TCP/IP network GNBD client GNBD client GNBD client GNBD Server Exported block device

GFS2 Global File System 2 One of cluster-aware file systems Multiple nodes can simultaneously access this filesystem Uses DLM(Distributed Lock Manager) of cman to maintain file system integrity OCFS is another cluster-aware file system

Global File System 2

One of cluster-aware file systems

Multiple nodes can simultaneously access this filesystem

Uses DLM(Distributed Lock Manager) of cman to maintain file system integrity

OCFS is another cluster-aware file system

GFS2 These nodes can access to the GFS2 file system simultaneously GNBD Server GFS2 GNBD client cman GNBD client cman GNBD client cman

DRBD Distributed Replicated Block Device RAID1 over a network Mirrors a whole block device over TCP/IP Available Active/Active with cluster file systems

Distributed Replicated Block Device

RAID1 over a network

Mirrors a whole block device over TCP/IP

Available Active/Active with cluster file systems

DRBD Replication Server Block Device Server Block Device

DM-MP Device-Mapper Multipath Bundles I/O paths to one virtual I/O path Can choose active/passive or active/active

Device-Mapper Multipath

Bundles I/O paths to one virtual I/O path

Can choose active/passive or active/active

DM-MP with SAN storage /dev/sda1 /dev/sdb1 Seen as one device /dev/mapper/mpath0 active/passive or active/active Node HBA1 HBA2 SAN swtich 1 SAN swtich 2 Storage CNTRLR1 CNTRLR2

A scalable storage system /dev/VG0/LV0 (CLVM) mount /dev/VG0/LV0 /mnt cman GNBD cman GNBD GNBD Server GFS2 GNBD Server GFS2 Replication (DRBD) /dev/mapper/mpath0 (DM-MP) /dev/gnbd0 /dev/gnbd1 GNBD Server GFS2 GNBD Server GFS2 Replication (DRBD) /dev/mapper/mpath1 (DM-MP) /dev/gnbd2 /dev/gnbd3

How to extend /dev/VG0/LV0 (CLVM) mount /dev/VG0/LV0 /mnt cman GNBD cman GNBD GNBD Server GFS2 GNBD Server GFS2 /dev/mapper/mpath0 /dev/gnbd0 /dev/gnbd1 GNBD Server GFS2 GNBD Server GFS2 /dev/mapper/mpath1 /dev/gnbd2 /dev/gnbd3 GNBD Server GFS2 GNBD Server GFS2 /dev/mapper/mpath2 /dev/gnbd4 /dev/gnbd5

I wonder ... Many components cause troubles? How about overhead and performance? How about stability? More better way? How about other than Red Hat Linux?

Many components cause troubles?

How about overhead and performance?

How about stability?

More better way?

How about other than Red Hat Linux?