Veritas Cluster Server (VCS) I/O Fencing parameters for racing (Solaris)

Article:TECH51519  |  Created: 2007-01-28  |  Updated: 2008-01-12  |  Article URL http://www.symantec.com/docs/TECH51519
Article Type
Technical Solution

Product(s)

Environment

Issue



Veritas Cluster Server (VCS) I/O Fencing parameters for racing (Solaris)

Solution



When communication between cluster nodes fails, causing the cluster to be divided into sub-clusters, these sub-clusters start a race to grab coordinator disks for data protection (VCS I/O Fencing). vxfen has a mechanism that enables cluster administrators to give larger sub-clusters better odds to win this race. This document describes the differences in implementation between VCS versions and their tunable parameters.

Note: While this mechanism can be used to give larger sub-clusters much better odds to win the race condition, it cannot be used to guarantee that the larger sub-cluster will always win.

1. How to give the odds
Prior to 4.1 MP2
If the number of nodes in a sub-cluster is less than the number of nodes leaving the original cluster, the sub-cluster repeats reading the coordinator disks to delay the start of the race. By default, the number of reads is calculated as cube of (the number of leaving nodes). For example, if a 5-node cluster is divided into a 3-node and a 2-node cluster,  the 2-node sub-cluster repeats reading coordinator disks 27 (= 3 cubed) times. A tunable parameter, max_read_coord_disk, can be used to change this value, as described later.

4.1 MP2 and 5.0 or later
If the number of nodes in a sub-cluster is less than the number of nodes leaving from the original cluster, the sub-cluster waits for a number of seconds before joining the race. This wait time is calculated as (the number of leaving nodes) x 5. For example, if a 5-node cluster is divided into a 3-node and a 2-node cluster,  the 2-node sub-cluster will wait for 15 (3 times 5) seconds. Tunable parameters min_delay and max_delay (4.1MP2), or vxfen_min_delay and vxfen_max_delay (5.0 or later), can be employed to change this wait time, as described later.

2. Tunable parameters
The following parameters can be specified in the file /kernel/drv/vxfen.conf. Use/tune these parameters only in situations where you often see a larger sub-cluster losing the vxfen race. Note that careful and ample testing is required to determine the most optimal values for a specific environment.

Prior to 4.1 MP2
max_read_coord_disk: The maximum number of times vxfen will loop reading coordinator disks. If the calculated repeat count exceeds this limit, this value will be used instead.
Default = 25
Min = 1
Max = 1000

4.1MP2
min_delay: The lower limit of the wait time in seconds. If the calculated wait time is below this limit, this value will be used instead.
Default = 1
Min = 1
Max = N/A

max_delay: The upper limit of the wait time in seconds. If the calculated wait time exceeds this limit, this value will be used instead.
Default: 600
Min = N/A
Max = 600

Limitations: The implementation of 4.1 MP2 is a subset 5.0 (or later) implementation and has some limitations. vxfen in 4.1 MP2 will "round down" the min_delay and max_delay values to a number that is a multiple of 5. For example, if the calculated delay time is 20 and max_delay specified is 18, the wait time value chosen will be 18. However, vxfen will only the wait for 15 seconds, ignoring the remaining 3 seconds. Therefore, to avoid confusion, it is recommended that only numbers that are a multiple of 5 be specified.

Note: With 4.1 MP2, the default and minimum values implemented for min_delay are a bit confusing, as they are both 1 and not a multiple of 5. Magic in the code prevents this value from being rounded down to 0, so specifying 1 here - or using the default value of 1 - is safe and will not be a problem.

VCS 5.0 or later allows more granularity,  so the delay can be specified in any number of seconds within the minimum and maximum boundaries.

5.0 or later
vxfen_min_delay: The lower limit of the wait time in seconds. If the calculated wait time is below this, this value will be used instead.
Default = 1
Min = 1
Max = 600

vxfen_max_delay: The upper limit of the wait time in seconds. If the calculated wait time exceeds this, this value will be used instead.
Default: 60
Min = the value of vxfen_min_delay
Max = 600


Supplemental Materials

SourceETrack
Value563541
Descriptionvxfen panics the larger cluster in a cascading split brain

SourceETrack
Value524705
Descriptionfencing shuts down the larger cluster after a splitbrain

SourceETrack
Value524044
DescriptionTwo node cluster loosing race with one node cluster.

SourceETrack
Value593089
Descriptionvxfen policies for time based delay


Legacy ID



288510


Article URL http://www.symantec.com/docs/TECH51519


Terms of use for this information are found in Legal Notices