Red Hat Cluster Suite / Handling of shutdown and mounting of GFS

[Ante Karamatic]

On Thu, 4 Sep 2008 23:55:47 +0200
Chris Joelly <chris-m-lists at joelly.net> wrote:

> The cluster with 2 nodes is running, but i don't know how to remove
> one node the correct way. I can move the active service (an IP
> address by now) to the second node and then want to remove the other
> node from the running cluster. cman_tool leave remove should be used
> for this which is recommended on the RH documentation. But if i try
> that i get the error message:
> 
> root at store02:/etc/cluster# cman_tool leave remove
> cman_tool: Error leaving cluster: Device or resource busy

Two node cluster in RHCS is a special case and people should avoid it.
I have one cluster with two nodes and I just hate it. Splitbrain is
very common (and only possible) in two node cluster.

> The only way to get out of this problem is to restart the whole
> cluster which brings down the service(s) and results in unnecessary
> fencing... Is there a known way to remove one node from the cluster
> without bringing down the whole cluster?

I've managed to bring one down, but as soon as it's up, whole rhcs get
unusable. Reboot helps :/

> Another strange thing comes up when i try to use GFS:
> 
> i have configured DRBD on a backing HW Raid10 device, use LVM2 to
> build a clusteraware VG, and on top of that use LVs and GFS across
> the two cluster nodes.
> 
> Using the GFS filesystems without noauto in fstab doesn't mount the
> filesystems on boot using /etc/init.d/gfs-tools. I think this is due
> to the ordering the sysv init scripts are started. All RHCS stuff is
> started from within rcS, and drbd is startet from within rc2. I read
> the section of the debian-policy to figure out if rcS is meant to run
> before rc2, but this isn't mentioned in the policy. So i assume that
> drbd is started in rc2 after rcS, which would mean that every
> filesystem on top of drbd is not able to mount on boot time... 
> Can anybody prove this?

I also use GFS on top of DRBD, and your observations are correct. But,
you really don't want DRBD started before GFS :D If there's a filesytem
on DRBD, DRBD client must be primary before you try to mount
filesystem. If this drbd client was out of sync for a long time,
becoming primary can take a while.

This is why I don't set up nodes to boot up automaticaly. I'd rather
connect to awaken node, start drbd sync and then manually mount
filesystem and start rhcs.

These things should be easier once we put upstart in use.

> The reason why i try to mount a GFS filesystem at boottime is that i 
> want to build cluster services on top of it, and that services (more 
> than one) are relying on one fs. A better solution would be to define
> a shared GFS filesystem resource which could be used across more than
> one cluster services, but the cluster take care that the filesystem is
> only mounted once...
> Can this be achieved with RHCS?

You can have same filesystem mounted on both nodes at the same time.
DRBD primary-primary + GFS on top of it.