Clusters

Sun Cluster and HA Data Service

Note: clustering can be configured for Storage, Apache, NFS, Oracle, Oracle RAC etc.

Prepare cluster nodes

1. Assign two servers, close proximity, OOB, LAN and SAN access as normal.
   Install Solaris10u11, enable remote root access (make life easy), install
   libintl, libgcc, libiconv, sudo, and findutils.
   Identify two available nics on one node, directly link the other nodes.
    $ dladm show-dev
    $ ifconfig e1000g2 plumb ; ifconfig e1000g3 plumb
2. Mount /globaldevices using a 512M slice (UFS, no volume mgt)
   Ensure multi-path support is configured,
    $ stmsboot -D fp -e
3. Ensure both hosts are in both /etc/hosts files
4. Generate RSA keys on both for passwordless auth as root, both ways
5. Ensure date is the same, ntp.conf is valid, and ntp daemon is enabled
6. Ensure RPC services are enabled,
$ svccfg -s rpc/bind setprop config/local_only = false
$ svccfg -s rpc/bind setprop config/enable_tcpwrappers = false
$ svcadm refresh rpc/bind
7. If using UFS and SVM, create state database replicas, and going forward ensure
   >50% are online.
8. Provision a 200M SAN LUN (quorum) and additional LUNs for data. Scan. Add label.
$ devfsadm -c disk ; cfgadm -lav ; format

Install Sun Cluster 3.2 Software (all nodes)

1. Download Java Availability Suite, extract it
2. Install the Sun Cluster software,
    $ cd /usr/local/src/suncluster_3.2u2/Solaris_sparc
    $ ./installer

  Choose CLI mode, agree to license, "No" to full Java ES, choose components
  5 and 7 (Cluster and Agent), package defaults, "No" to Multilingual
  support, confirm component upgrades, choose to configure later, finally
  "No" to update notifications.

Create the node cluster

1. Check your direct connected network interfaces are plumbed.
2. Open a console connection to each node and login.
   On node-1, run scinstall, new cluster, new cluster, custom,
   enter cluster name, specify nodes, no DES authentication, use two networks,
   cluster does not use switches, specify local and remote adapters,
   accept default net address and netmask, don't turn off global fencing,
   disable automatic quorum device selection, use /globaldevices on both,
   create cluster, dont interrupt cluster creation.
   Cluster check occurs, remote config is applied, and then node reboots.
   Note: Do this to completely start over:
   $ reboot -- -x
   $ scinstall -r
   $ /var/sadm/prod/SUNWentsyssc32u2/uninstall
   $ reboot -- -r
3. Check services, check 2 nodes online, 2 transport paths online, check for the
   new global device mountpoint,
$ scstat
$ df -h
$ ps -ef | grep -i clust
$ cluster status
4. Reboot one at a time if errors are reported.
5. Identify the disk devices available to the cluster,
$ cldev populate
$ scdidadm -L
$ echo | format
6. Confirm cluster install mode is still set to "enabled"
$ cluster show -t global | grep installmode
7. Add the shared quorum disk, and then exit from "installmode",
clsetup > add quorum? yes > device type? direct attached shared disk >
which device? eg. d7 > reset installmode? yes
$ clquorum status
$ clquorum list -v
$ cluster show -t global | grep installmode

* Note, quorum disks can be added or replaced (add, remove, reset)

Add local data disks to primary node

1. Review the existing SVM configuration (create state database replicas if none exist)
$ metadb -i | grep [0-9] ; metastat -p
2. If not already done, provision SAN storage, scan, label and re-populate device tree,
$ devfsadm -c disk ; cfgadm -lav ; format
$ cldev populate
$ scdidadm -L
3. Create SVM diskset associating cluster nodes and a shared disk,
$ metaset -s data -a -h node1 node2
$ metaset -s data -a /dev/did/rdsk/d6
$ metaset -s data
4. Create SVM volume within diskset, create the directory
$ metainit -s data d101 1 1 /dev/did/rdsk/d6s0
$ metastat -s data -p
<optionally create mirrored metadevice>
5. Write a filesystem, mount the volume, repeat for additional volumes,
$ mkdir /data
$ echo "/dev/md/data/dsk/d101 /dev/md/data/rdsk/d101 /data ufs 2 no -" >> /etc/vfstab
$ newfs /dev/md/data/rdsk/d101
$ mount /data

* Note: the disk is not shared in the cluster yet

Configure cluster storage resources for HA

1. Register the HA storage resource type,
$ clrt register SUNW.HAStoragePlus
$ clrt list
2. Configure HA storage for each filesystem using wizard,
clsetup > data services > HA storage > review node selection > choose both storage types >
select mount point (eg /data) > select SVM/global device group > review resource names >
create configuration and quit.
3. Repeat with the wizard for other filesystems.
4. Review resources and resource groups,
$ clrg status
$ clrs status
$ cldg status
$ cluster show -t resource,resourcegroup

* Note, can separate or group the resources together.

Fail-over a storage resource group

1. Review status of resource groups, and mount status,
$ clrg status ; clrs status ; df -h /data
2. Failover a resource group to other node, review the change,
$ clrg switch -n node2 data-rg
$ df -h | grep /data
* /data is mounted on the other node now. Fail-back if you wish.

Patching with minimal downtime

1. Extract the Solaris patches on each server into /var/tmp, remove the zip file
2. On the inactive node console as root, evacuate resources,
$ cluster status
$ df -h | grep data
< resources not mounted here, evacuate just to make sure >
$ clnode evacuate `hostname`
$ cluster status
3. Same session, go to single-user non-cluster mode, patch, and then reboot
$ reboot -- -sx
$ cd /var/tmp/10* ; ./installcluster --s10patchset
$ reboot
    < monitor /var/adm/messages on the primary, wait until we rejoin cluster >
$ tail -f /var/adm/messages
4. On the active node console as root, check status, stop apps (if applies), failover all resources,
$ clrs status ; clrg status ; cluster status
< application outage begins here, resources go to other node >
$ clnode evacuate `hostname`
$ clrs status ; clrg status
$ df -h | grep data
$ ssh node2 "df -h" | grep data
< application outage ends when volumes are mounted and apps started >
5. Same session, repeat actions of step 3 for patching, reboot and rejoin cluster,
$ clrg status ; clrs status ; cluster status
$ df -h | grep data
$ ssh node2 "df -h" | grep data
6. Optionally, failback resources to node1 as the primary
$ clrg switch -M -n node1 data-rg

Replace quorum disk

1. Add infrastructure and paths if required. Provision the quorum 200M disk to both
   nodes. Evacuate and reboot to see the storage subsystem and disk if required, an
   outage may be necessary here. If the storage subsystem is not changing, just scan
   for the new disk without any evacuations and reboots,
$ devfsadm -c disk ; cfgadm -lav  
2. Configure the new disk in Solaris (label, partition table), add to global devices,
$ format
$ cldev populate
$ cldev status
$ scdidadm -L
3. Replace the quorum disk,
$ clq add d16
$ clq status
$ clq remove d7
$ clq status
4. Unprovision the original disk, wait a few minutes, cleanup the device list,
$ cfgadm -lav
$ devfsadm -C
$ cldev clear -v
$ cldev status
< disk may report status of "Fail" for a while >

Migrate data to new array

1. Add infrastructure and paths if required. Provision the data disk to both nodes.
   Evacuate node2, reboot it to see the new storage subsystem and disk. Wait for it
   to re-join the cluster. Repeat for node1.
2. Add disk label and partition table. Add to global devices,
$ format
$ cldev populate
$ scdidadm -L
3. Create the new SVM disk set and associate the disk,
$ metaset -s data2 -a -h node1 node2
$ metaset -s data2 -a /dev/did/rdsk/d14
$ metaset -s data2
4. Create the SVM volume in the new diskset, write the UFS filesystem,
$ metainit -s data2 d102 1 1 /dev/did/rdsk/d14s0
$ metastat -s data2 -p
$ newfs /dev/md/data2/rdsk/d102
5. Remove original data volume from cluster, this will also un-mount it,
$ clrg delete -F data-rg
6. Edit /etc/vfstab on both nodes, change the original mount, mount both original and new,
$ vi /etc/vfstab
$ mount /data_orig
$ mount /data
< Note, /data is empty, it is the new storage >
7. Create the resource groups and associate resources,
$ clrg create data-rg
$ clrs create -t SUNW.HAStoragePlus:6 -g data-rg \
 -p FilesystemMountPoints=/data -p Affinityon=true data-rs
$ clrg manage data-rg
$ clrg online data-rg
$ cluster status
$ df -h | grep data
8. Migrate data from /data_orig/ to /data/,
$ cp -rp /data_orig/* /data/
$ umount /data_orig
$ df -h | grep data
9. Cleanup and remove configuration for /data_orig,
$ metastat -s data -p
$ metaclear -s data -a
$ metaset -s data -P
$ ls /dev/md
$ cfgadm -lav
$ devfsadm -C
$ cldev clear -v
$ cldev status
$ echo | format
< Test resource group fail-over works >
$ clnode evacuate `hostname`
< If required cleanup devices with reboot >
$ touch /reconfigure ; reboot

MISC

$ clrg online -M rgname # manage and online the rg
$ clrg switch -M -n nodename rgname # manage rg on specified node
$ clrg offline rgname # offline, but still managed rg
$ clrs disable rsname # disable resource

Refer also official Oracle doco and this handy link,
 https://www.princeton.edu/~unix/Solaris/troubleshoot/suncluster.html