Red Hat Global File System (GFS)


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Red Hat Global File System (GFS)

  1. 1. Red Hat Storage Cluster (GFS, CLVM, GNBD) GFS allows multiple nodes to share storage at a block level as if thestorage were connected locally to each cluster node. Schubert Zhang, Guangxian Liao Jul, 2008 1
  2. 2. Deployment GFS node GFS node GFS node Applications Applications Applications GFS GFS GFS /mnt/gfs /mnt/gfs /mnt/gfs CLVM/LVM CLVM/LVM CLVM/LVM vg-lv vg-lv vg-lv GNBD Client GNBD Client GNBD Client import import import /dev/gnbd/x /dev/gnbd/x /dev/gnbd/x TCP/IP Network Storage Level GNBD Server GNBD Server export export Block Devices Block Devices Disks... Disks... GNBD Server Node GNBD Server Node“Economy and Performance” deployment. 2
  3. 3. Architecture• Pay attention to the relationship of cluster software components. – Cluster Infrastructure (Common): CMAN, DLM, CCS, Fencing – GNBD: Server and Client – CLVM – GFS 3
  4. 4. Cluster Infrastructure• We only concern cluster type for storage.• Necessary infrastructure components: – Cluster management (CMAN) • It’s a distributed cluster manager and runs in each cluster node. • Keeps track of cluster quorum, avoids "split-brain". (Like chubby’s Paxos Protocol?) • Keeps track of membership. • library, cman_tool • dlm_controld: started by cman init script to manage dlm in kernel • gfs_controld : started by cman init script to manage gfs in kernel • groupd: started by cman init script to interface between openais/cman and dlm_controld/gfs_controld/fenced , group_tool – Lock management (DLM) • To synchronize access to shared resources (shared storage, etc.). • Runs in each cluster node. • library. • GFS and CLVM use locks from DLM. • GFS uses locks from the lock manager to synchronize access to file system metadata (on shared storage). • CLVM uses locks from the lock manager to synchronize updates to LVM volumes and volume groups (also on shared storage). • Like chubby’s lock function? 4
  5. 5. Cluster Infrastructure (cont.)– Cluster configuration management (CCS) • Runs in each cluster node. (ccsd) • Synchronization/up-to-date of cluster configuration file. Propagates modification in a cluster. • Other components (eg. CMAN) access configuration info via CCS. • /etc/cluster/cluster.conf (Cluster Name, Cluster nodes, Fence, Resources, etc.) • ccs_tool: to make online updates of CCS configuration files • ccs_test: to retrieve information from configuration files through ccsd.– Fencing • Fencing is the disconnection of a node from the clusters shared storage. Fencing cuts off I/O from shared storage, thus ensuring data integrity. The cluster infrastructure performs fencing through the fence daemon, fenced. • GNBD fencing: fence_gnbd? • fence_tool • cluster configuration file: fencing-method, fencing agent, fencing device for each node in the cluster. 5
  6. 6. GNBD• An ancillary component of GFS that exports block-level storage to Ethernet.• Global Network Block Device – GNBD provides block-device access for Red Hat GFS over TCP/IP. GNBD is similar in concept to NBD; – GNBD is GFS-specific and tuned solely for use with GFS.• Two major components – GNBD Server – GNBD Client 6
  7. 7. GNBD (server)• Exports block-level storage from its local storage.• gnbd_serv process – GNBD server need not join the cluster manager.• gnbd_export – Export block devices. 7
  8. 8. GNBD (client)• A GNBD client runs in a node with GFS.• Imports block device exported by GNBD server.• Multiple GNBD clients can access a device exported by a GNBD server, thus making a GNBD suitable for use by a group of nodes running GFS.• gnbd.ko – A kernel module• gnbd_import – Import remote block devices form GNBD server. 8
  9. 9. CLVM• Provides volume management of cluster storage.• A cluster-wide version of LVM2• CLVM provides the same capabilities as LVM2 on a single node, but makes the logical volumes created with CLVM available to all nodes in a cluster.• CLVM uses the lock-management service provided by the cluster infrastructure.• Using CLVM requires minor changes to /etc/lvm/lvm.conf for cluster- wide locking.• clvmd: – A daemon that provides clustering extensions to the standard LVM2 tool set and allows LVM2 commands to manage shared storage. – Runs on each cluster node. – Distributes LVM metadata updates in a cluster, thereby presenting each cluster node with the same view of the logical volumes 9
  10. 10. CLVM (cont.)• lvm: LVM2 command line tools.• /etc/lvm/lvm.conf• pvcreate – block devices/partitions -> PV• vgcreate – PV(s) -> VG• lvcreate – VG -> LV(s)• Ready for make file system on LV. 10
  11. 11. GFS• To simultaneously access a block device that is shared among the nodes.• Single, consistent view of the FS name space across GFS nodes in a cluster.• Native FS under VFS, POSIX interface to applications.• Distributed metadata and multiple journals.• Uses lock manager to coordinate I/O.• When one node changes data on a GFS file system, that change is immediately visible to the other cluster nodes using that file system.• Scale the cluster seamlessly by adding servers or storage on the fly.• We use a “Economy and Performance” deployment. 11
  12. 12. GFS (cont.)• gfs.ko: kernel module, loaded on each GFS cluster node.• gfs_mkfs: create a GFS on a storage device.• gfs_tool: configures or tunes a GFS.• gfs_grow: grows a mounted GFS.• gfs_jadd: adds journals to a mounted GFS.• gfs_quota: manages quotas on a mounted GFS.• gfs_fsck: repairs an unmounted GFS.• mount.gfs: mount helper called by mount. 12
  13. 13. Fencing• We must configure each GFS node in the cluster for at least one form of fencing. 13
  14. 14. Setup a Cluster (prepare)• Software Installation – Install default packages of “Clustering” and “Storage Clustering” in each node. – The rpms in cdrom: /ClusterStorage and /Cluster – cman’s rpm in cdrom: /Server – Major RPM (if need dependence, install depended rpm) • cman-2.0.60-1.el5.i386.rpm • modcluster-0.8-27.el5.i386.rpm • gnbd-1.1.5-1.el5.i386.rpm • kmod-gnbd-0.1.3- • kmod-gfs-0.1.16- • lvm2-cluster-2.02.16-3.el5.i386.rpm • Global_File_System-en-US-5.0.0-4.noarch.rpm • gfs-utils-0.1.11-1.el5.i386.rpm • gfs2-utils-0.1.25-1.el5.i386.rpm • etc.• Network – Disable firewall and SELinux – Enable multicast and IGMP. – Configure /etc/hosts or DNS for hostname (import!)• Machine hostnames – test1 (gnbd server) – test2 (gfs node) – test3 (gfs node) – test4 (gnbd server) 14
  15. 15. Setup a Cluster (GNBD server)• In GNBD server node, need not start cman, i.e., the GNBD server is not a member of the cluster.(1) Start GNBD server process (man gnbd_serv) # gnbd_serv -n(2) Export block device (man gnbd_export) # gnbd_export -v -d /dev/sda3 -e gnbdnode1 –c Note1: must enable cache since no cman. Note2: the block device should be a disk partition (how about LV? The document says LV is not supported).(3) Check the export # gnbd_export -l(4) Add to /etc/rc.local gnbd_serv –n gnbd_export -v /dev/sda3 -e gnbdnode1 -c 15
  16. 16. Setup a Cluster (cluster infrastructure)• Initially configure a cluster – /etc/cluster/cluster.conf: generated by “system-config-cluster” or manually. (only config clustername and node members) – In one GFS node, use “system-config-cluster” to create a new cluster(name:cluster1) and add a node (name: test2) – The cluster.conf <?xml version="1.0" ?> <cluster alias="cluster1" config_version="5" name="cluster1"> <fence_daemon post_fail_delay="0" post_join_delay="3"/> <clusternodes> <clusternode name="test2" nodeid="1" votes="1"> <fence/> </clusternode> </clusternodes> <cman expected_votes="1" two_node="1"/> <fencedevices/> <rm> <failoverdomains/> <resources/> </rm> </cluster>• Start infrastructure components – service cman start (refer to /etc/init.d/cman) • Load kernel modules (configfs, dlm, lock_dlm) • Mount configfs (I think it like chubby’s files ystem space) • Start ccsd daemon • Start cman (no daemon, use cman_tool join to join this node to cluster) • Start daemons (start groupd, fenced, dlm_controld, gfs_controld) • Start fencing (start fenced daemon, and use fence_tool to join this node to fence domain) 16
  17. 17. Setup a Cluster (GNBD client)• Load kernel module gnbd.ko # echo “modprobe gnbd” >/etc/sysconfig/modules/gnbd.modules # chmod 755 /etc/sysconfig/modules/gnbd.modules # modprobe gnbd Then, the gnbd.ko will be loaded when the node boot-up.• Import GNBD # gnbd_import -i test1 Then, we can find a block device /dev/gnbd/gnbdnode1, and it is same as /dev/gnbd0. create a /etc/init.d/gnbd-client as a service script # chmod 755 /etc/init.d/gnbd-client # chkconfig --add gnbd-client Since gnbd_import -i should be done early then clvmd and gnbd_import –R should be done later then clvmd, we assign special start-number (23<24) and stop-number(77>76) in script /etc/init.d/gnbd-client. Thus, the gnbd will be imported automatically when node boot-up. 17
  18. 18. Setup a Cluster (CLVM)• Start clvmd # service clvmd start # chkconfig --level 35 clvmd• pvcreate # pvcreate /dev/gnbd0, or # pvcreate /dev/gnbd/gnbdnode1 Use lvmdiskscan or pvdisplay or pvscan to display status.• vgcteare # vgcreate vg1 /dev/gnbd0 Use vgdisplay or vgscan to display status.• lvcreate # lvcreate -l 100%FREE -n lv1 vg1 Use lvdisplay and lvscan to display status.• Restart clvmd # service clvmd rerestart clvmd is responsible for sync the lvm configuration among the cluster nodes. Now, we can find a new logical volume block device @ /dev/vg1/lv1, and we can make file system on it. 18
  19. 19. Setup a Cluster (GFS)• Make sure that the clocks on the GFS nodes are synchronized. (Use NTP)• Make GFS file system # gfs_mkfs -p lock_dlm -t cluster1:testgfs -j 4 /dev/vg1/lv1 Note: -j number same node number. One journal is required for each node that mounts a GFS file system. Make sure to account for additional journals needed for future expansion.• Mount GFS # mkdir /mnt/gfs (1) # mount -t gfs -o acl /dev/vg1/lv1 /mnt/gfs (2) add to /etc/fstab /dev/vg1/lv1 /mnt/gfs gfs defaults,acl 0 0 # mount -a -t gfs # chkconfig --level 35 gfs (refer to /etc/init.d/gfs) Now, the GFS file system is accessible @ /mnt/gfs/ 19
  20. 20. Setup a Cluster (add a new GFS node)• Install packages in the new node• Add new node information in cluster.conf in a existing node. – Use ccs_tool addnode, or – Use system-config-cluster• Copy (scp) the /etc/cluster.conf to new node.• Import GNBD on the new node• Stop components on all running node (if there are more than 2 node existing, need not do this step) # service gfs stop # service clvmd stop # service cman stop• Then start components on all running node and the new added node # service cman start # service clvmd start # service gfs start The clvmd will sync the metadata of logical volumes to the new added node, so, when clvmd is started, the /dev/vg1/lv1 will be visible on the now node. 20
  21. 21. Setup a Cluster (add a new GNBD node)• Setup a new GNBD server (machine test4) # gnbd_import -l and export a new block device. Device name : gnbdnode1 # gnbd_serv -n ---------------------- # gnbd_export -v -d /dev/sda3 -e gnbdnode2 –c Minor # : 0 # gnbd_export -l sysfs name : /block/gnbd0• Import the new GNBD (on all of the cluster Server : test1 nodes) Port : 14567 # gnbd_import -i test4 State : Open Connected Clear # gnbd_import –l Readonly : No• Make a new PV (on one of the cluster node) Sectors : 3984120 # pvcreate –v /dev/gnbd1 Then, we can find the new PV on all nodes in the cluster, by pvdisplay or lvmdiskscan or pvscan Device name : gnbdnode2 or pvs. ----------------------• Extend the VG (on one of the cluster node) Minor # : 1 # vgextend -v vg1 /dev/gnbd1 sysfs name : /block/gnbd1 Then, we can find the extended VG and changed PV on all nodes in the cluster, by vgdisplay or Server : test4 vgfscan or vgs. Port : 14567• Extend the LV (on one of the cluster node) State : Close Connected # lvextend -v -l +100%FREE Clear /dev/vg1/lv1 Then, we can find the extended LV on all nodes in Readonly : No the cluster, by lvdisplay or lvfscan or lvs. Sectors : 4273290• Grow the GFS (on one of the cluster node) (next page) 21
  22. 22. Setup a Cluster (grow the GFS)• The gfs_grow command must be run on a mounted file system. Only needs to be run on one node in a cluster.• Grow # gfs_grow /mnt/gfs• Sometimes after gfs_grow, df and lvdisplay are hung. Need reboot the system. 22
  23. 23. Evaluation• Size – GFS is based on a 64-bit architecture, which can theoretically accommodate an 8 EB file system. However, the current supported maximum size of a GFS file system is 25 TB. (But there is a note in Red Hat document: “If your system requires GFS file systems larger than 25 TB, contact your Red Hat service representative.”)• Essential benchmarks – Refer to GlusterFS’s benchmarks: ks• Iozone benchmarks – Refer to• Lock test – Always blocks the process when it set a new lock which conflict with other process. 23
  24. 24. Conclusions• The cluster infrastructure is too complex and weak, sometimes fail (such as cman, fencing, etc).• The GNDB is simple and robust, but lacks flexibility.• The CLVM is ok, but too complex to use.• The GFS is very weak, sometime fail. (mount is ok, but umount often fail)• The “two levels” (GNBDStorage+GFSCluster) deployment is not meet the “cloud” goal.• Not easy to add a new GFS cluster node or a GNBD storage node.• No data replicas for safety.• Risks when a GNBD node fails. When one GNBD node fails, the data GFS is not accessible. The Red Hat Cluster solution is not based on the Assumptions like GoogleFS and GlusterFS, (i.e. “The system is built from many computer that often fail.”), it is not easy and not safe to use in a moderate scale cluster. So, I think the Red Hat Storage Cluster is not a good designed solution, and has no good future. 24