Latest breaking news for Veritas Storage Solutions and Veritas Cluster File Solutions 4.0 products for AIX

Article:TECH33563  |  Created: 2010-01-19  |  Updated: 2010-01-19  |  Article URL http://www.symantec.com/docs/TECH33563
Article Type
Technical Solution

Product(s)

Environment

Issue



Latest breaking news for Veritas Storage Solutions and Veritas Cluster File Solutions 4.0 products for AIX

Solution





End-of-Availability and End-of-Support for Veritas Storage Foundation and High Availability Solutions 4.0 (AIX)

The End-of-Availability and End-of-Support Announcement for Veritas Storage Foundation and High Availability Solutions 4.0 (AIX) is available at http://entsupport.symantec.com/docs/283963 


This TechNote provides late breaking information regarding all Veritas Storage Solutions and Veritas Cluster File Solutions (tm) 4.0 products on AIX.  Symantec Corporation updates this TechNote as new customer related information, pertaining to 4.0 products, becomes available. Symantec recommends that you frequently check this TechNote for updates.

4.0 Related Issues
Patches
Symantec has uncovered several issues with the AIX Storage Foundation 4.0 products through continual testing.  While these issues will not impact the majority of customers, all customers should evaluate their environments with respect to each issue listed below, and determine if they are at risk. After reviewing the provided information, you can contact Symantec Enterprise Technical Support for additional questions or to obtain a patch.

Note: Updating the Storage Solutions and Cluster File Solutions 4.0 base products at MP1 level will correct the 4.0 related issues:

1. Patch to Veritas File System  - VRTSvxfs at 4.0.0.10 revision level
The following two defects were discovered and resolved in the VERITAS File System  4.0 release. The point patch for VRTSvxfs at revision 4.0.0.10 is available for applying the fixes for the defects.

Defect 1.AIX vx_vmm_buf_count too high on 32-bit kernel - can exhaust kernel heap
Description:
The AIX kernel allows allocation of pinned memory up to 480MB in installations booted with 32-bit AIX kernel, regardless of the size of physical memory. The VERITAS File System tunable, vx_vmm_buf_count, is auto tuned based on the available physical memory size, with no regard to the 32-bit kernel heap restriction. The result of this tuning is a large number of Virtual Memory Manager (VMM) buffers causing over-allocation and wasting memory. With the application workload using a large amount of memory, this wasting of memory must be avoided. Storage Foundation installations in a 32-bit kernel environment can install this patch to avoid consuming excess kernel pinned memory.

Defect 2. Storage Foundation Cluster File System, stress test incurredvx_iaccess:1a
Description:
As a result of a very small race condition, insufficient access checking was performed during an attempt to create an existing file with non-exclusive access permission bits. With such an attempt, the existing file is recreated, instead of being denied. The failure of checking access and the denial of access occur only when the process attempting to create is the owner of the existing file.

2. IBM/AIX patch necessary for VERITAS Volume Manager (tm)
A defect at scsidisk layer was discovered in AIX 5.2 at ML03 and ML04. The defect causes the VERITAS Volume Manager configuration daemon (vxconfigd) to hang during boot process. To resolve this issue, you must download and install the APAR IY59620. This APAR is available at the IBM Support ftp site.

Before the availability of the APAR, IBM had made AIX e-fixes temporarily available to address this issue. If you had already installed the IBM/AIX e-fixes, you should remove them and install the APAR instead. The old e-fixes located at the following IBM support site were52FIY59620.epkg.Z (for AIX 5200-3) and 52HIY59620.epkg.Z (for AIX 5200-04).
     ftp://service.software.ibm.com/aix/efixes/thrpgio/

 3. Patch to VERITAS Oracle Disk Manager (ODM) - VRTSodm at 4.0.0.10 revision

Until Oracle version 9iR2, Oracle always skipped the first 4K bytes of a RAW LVM. The Oracle header block starts at offset 4K of the RAW volume.  To make Volume Manager RAW volumes compatible with Logical Volume Manager (LVM), Oracle also skips the first 4K of a Volume Manager RAW volume.  For databases having  db_block_size larger than 4K, this causes misaligned I/O at the stripe boundary of a striped volume. For example, for database with a db_block_size of 16K and a RAW volume with stripe layout and stripe width 32K, every other db_block spans two LUNS/disks.

To solve this misaligned I/O issue, in 9iR2 AIX introduced a new type of volume with devsubtype DS_LVZ. This kind of LVM is also called a "z" type volume. From 9iR2 onwards, Oracle started recognizing these new "z" type volumes and does not skip the first 4K bytes in the volume.

VERITAS also enhanced Volume Manager and introduced a new type of volume with devsubtype as DS_VMZ. This kind of volume is also termed "o" type Volume Manager volume. Oracle is enhancing both 9iR2 and 10g to recognize this new type of Volume Manager volume. A patch from Oracle will be available soon that needs to be applied to 9.2.0.5 and 10.1. The reference bug number for this Oracle patch is 3712203.

This ODM patch recognizes DS_VMZ type RAW Volume Manager volumes. Without this patch, when ODM is enabled, the Oracle instance will fail with error "ORA-01251: Unknown File Header Version read".

4.0 MP1 Related Issues
Patches
VERITAS has uncovered several issues in conjunction with the Storage Solutions and Cluster File Solutions 4.0 MP1 products. While these issues will not impact the majority of our customers, we ask all customers to evaluate their environments with respect to each issue listed below, and determine if they are at risk. After reviewing the provided information, you can contact VERITAS Support for additional questions or to obtain more information.

1. There are critical fixes for AIX which are required for supporting Storage Solutions and Cluster File Solutions 4.0MP1 products on AIX 5.1, 5.2, and Storage Foundation Products on AIX 5.3. Those fixes can be downloaded and installed before installing or upgrading Storage Foundation products to MP1 level. Refer to the product release notes for AIX operating system requirements.

For AIX 5.1
The minimum requirement is ML6 with APAR IY67577 and IY65263.  Both these APARs are available for download and install.

For AIX 5.2
The minimum requirement is ML4 with APARs IY67584, IY65717 and IY59620 and all these APARs are available for download and install.

For AIX 5.3
The minimum requirement is ML1 with APARs IY66534, IY65459, IY65469, IY68510, and IY68366.  All these APARs are available for download and install.
For Storage Foundation Cluster File System, Storage Foundation for Oracle Real Application Clusters, or Cluster Server, install the following:  APAR IY68366.

2. Please read this when you upgrade your system to AIX 5.3 and are running 4.0MP1 release
  ############################################################################

Symantec has uncovered a critical problem with VERITAS File System on AIX 5.3. This issue affects all VERITAS Storage Foundation Products on AIX 5.3. Storage Foundation Products on AIX 5.1 or 5.2 are not affected.
Possible symptoms of this issue are: failure to perform I/O to files or failure to start up an instance of Oracle or DB2 database.
The problem appears as soon the base operating system filesets of AIX, bos.mp or bos.mp64, are updated to 5.3.0.31 level or higher. Installing certain APARs, such as IY69000, or updating AIX 5.3 to ML03 updates the base operating system fileset level to 5.3.0.31 or higher.

Symantec has issued a fix for this problem. Before installing the fix,  you must determine whether the Storage Foundation Product in your AIX 5.3 environment requires the patch or not. To check this you can run the lslpp command, as shown below, and check the fileset level of the latest bos.mp or bos.mp64 installed on your system.
# lslpp -l bos.mp bos.mp64

If the displayed output shows the level below 5.3.0.31, you don't have to install the fix.
Storage Foundation Products version 4.0 MP1 on AIX 5.3 are affected by this problem. Symantec has released a patch to resolve this problem. To determine if the patch is applicable to your Storage Foundation installation, you must run the lslpp command, as shown below, to determine the version of VERITAS File System installed on your system:
# lslpp -l VRTSvxfs

For VRTSvxfs at 4.0.1.x level (for any x), the installed version of Storage Foundation is 4.0 MP1, and you must install VRTSvxfs.4.0.1.2-incr.bff
Contact Symantec Support to obtain the required patch.

Errors in Product Release Notes
1. Several of the VERITAS Product Release Notes (CFS,SF,SFO,SFDB2,GSG) incorrectly specify that APAR IY65459 needs to be installed to meet minimum operating system requirements. The correct APAR is IY65469.
2. Release Notes for VERITAS Storage Foundation and VERITAS Storage Foundation Cluster File System state to execute "df -k | grep vxfs" to check for mounted VxFS file systems or checkpoints. This command is incorrect. The correct command is "mount | grep vxfs".
3. Release Notes for VERITAS Storage Foundation Cluster File System for AIX 5.3 suggest following steps 6, 7 and 8 in the section titled "Installing the Product". Those steps are not necessary. Instead, refer to Installation and Administration Guide of VERITAS Storage Foundation Cluster File System 4.0 product, and follow steps 16 - 30 specified in that Installation Guide document.

Other issues
1. VERITAS Etrack Incident 263556: vxdctl may be slow with EMC Clariion CX arrays
The EMC Clarion CX arrays in a multinode, multipath Cluster Volume Manager  environment may be susceptible to slow device discovery due to the slowness of the array to process key registration commands. This may be observed when issuing the vxdctl enable or scandisk commands as well as device discovery during system boot up. This is not an issue in 4.0 MP2 and 4.0 MP3.

2. VERITAS Etrack Incident e315766
The VERITAS Cluster Server Simulator can terminate on Cluster Server 4.0 MP1 for AIX. The problem occurs in a simulated configuration when users enable a service group and attempt to switch it to another system.

3. VERITAS defect 152676
When uninstalling Storage Foundation for Oracle/RAC or Storage Foundation Cluster File System,  the uninstall process may hang on one of the nodes. The result is failure to uninstall due to the hang on one of the nodes in the cluster. The following is the log output from uninstallsfcfs script, which shows the failure to stop Cluster Manager and Agents on node4 of the cluster:
Are you sure you want to uninstall SFCFS filesets? [y,n,q] (y)
 Stopping Cluster Manager and Agents on node1 ...................... Done
 Stopping Cluster Manager and Agents on node2 ...................... Done
 Stopping Cluster Manager and Agents on node3 ...................... Done
 Stopping Cluster Manager and Agents on node4
The failure is caused by the hang of the following command on node4:
  /opt/VRTS/bin/fsclustadm cfsdeinit

Related TechNote for AxRT 4.0
VERITAS Cluster Server:  http://support.veritas.com/docs/276074
Related TechNotes for AxRT 4.0 MP1:
Storage Foundation for Oracle RAC: http://support.veritas.com/docs/275287
Storage Foundation for Oracle: http://support.veritas.com/docs/275816  

4.0 MP2 Related Issues
Patches
Critical fixes for AIX are required for supporting VERITAS Storage Solutions products (Volume Manager, VERITAS File System, Volume Replicator, VERITAS Cluster Server, Storage Foundation for DB2, Storage Foundation for Oracle) and Cluster File Solutions products (Storage Foundation RAC, Storage Foundation Cluster File System) at the 4.0 MP2 level on AIX 5.1, 5.2, and AIX 5.3. The fixes should be downloaded and installed before installing or upgrading Storage Solutions and Cluster File Solutions products at the 4.0 MP2 level. Refer to each product release notes for additional information.
Navigate to the IBM Quick links for AIX fixes:
http://www-1.ibm.com/servers/eserver/support/pseries/aixfixes.html
On the page, in the Specific fixes section, click a specific O/S link. From the pull-down menu, select the Search by APAR number or abstract field, and enter the fix number in the search string field

For AIX 5.1
The minimum requirements are ML6 with APAR IY67577 and IY65263. The APARS are available for download and installation.

For AIX 5.2
The minimum requirements are ML4 with APARS IY67584, IY65717 and IY59620. The APARS are available for download and install.

For AIX 5.3
The minimum requirements are ML2 with APARS IY65469, IY68510, IY69539, IY70336, and IY73101. These APARS are available for download and installation.
Before the general availability of these APARs, IBM had made AIX e-fixes temporarily available to address issues for APARs IY69539, IY73101 and IY68989 (required by Oracle, not Symantec). If you have already installed the IBM/AIX e-fixes, you should remove them and install the APARs instead.

For MultiNICB users
For AIX 5.3,  the minimum requirements are ML2 with APAR IY65664. The APAR is available for download and installation.

Storage Foundation for Oracle/RAC with Cluster Volume Manager and VERITAS Volume Replicator
The following steps are needed to use Volume Replicator on Storage Foundation for Oracle RAC.

Packages VRTSvrpro, VRTSvrw, and VRTSvrdoc have to be installed on the system before the upgrade to VERITAS Storage Foundation for Oracle RAC software to the MP2 level with Volume Replicator is enabled. Customers with an existing installation of VERITAS Storage Foundation for Oracle RAC 4.0 must install Volume Replicator 4.0 packages from the Symantec FTP site ftp://ftp.support.veritas.com :
1. Downloadvvr.4.0.00.0.aix.tar.gz
2. Unzip and untar the this tar file
3. Change the directory to/volume_replicator/pkgs
4. Install the Volume Replicator packages: VRTSvrpro, VRTSvrw, and VRTSvrdoc
5. Insert the VERITAS Cluster File Solution 4.0 MP2 software disc
6. Run the installmp command to install maintenance patches for both Volume Replicator and Storage Foundation for Oracle/RAC
Refer to the VERITAS Storage Foundation for Oracle RAC Installation and Configuration Guide for details on upgrading

Limitation with RDAC driver and FAStT array for coordinator disks
AIX uses the RDAC driver for FAStT arrays for multipathing to connected storage. Since it is an active/passive array, only the current active path is exposed to clients. The I/O fencing driver, vxfen, can use only a single active path and has no prior knowledge of the passive paths to the coordinator disks on an array. Therefore, if the single active path fails, all nodes in the cluster lose access to the coordinator disks. The loss of the path to the coordinator disks can potentially go unnoticed until a reboot, split brain, or any other reason that leads to a cluster membership change. In any of these conditions, the cluster cannot form, and all nodes panic to prevent data corruption. No data loss occurs.

DMP Support of FastT Devices (RDAC):
AIX uses the RDAC driver for FAStT arrays for multipathing to connected storage. Storage Foundation coexists with the RDAC driver on AIX and for Storage Foundation to claim the FAStT devices, the RDAC driver must be installed on the machine. If the RDAC is uninstalled from the machine, Storage Foundation will not be able to claim FAStT array devices. In addition, AVT mode should be disabled for the ASL to properly claim devices.

VERITAS Volume Manager and dynamic multipathing (DMP) issues in conjunction with AIX (FastFail Feature)
DMP failover takes significant time when the path is disabled from the switch or array side in a SAN environment. Path failover can be very slow if a cable is pulled from the switch or array side,  but not from the host side. AIX implemented a feature in the 5200-01 (and higher) update that prevents the long failover time and is called "Dynamic Tracking". A fastfail attribute can be  set on every controller which would fix the original issue. IBM also released this feature as part of the 5100-06 update.
Following is the example for setting up/changing these parameters on AIX 5200-02. If Volume Manager is already installed and configured, perform the following:
# chdev -l fscsi0 -a fc_err_recov=fast_fail -P
# chdev -l fscsi0 -a dyntrk=yes -P
and then reboot the system
To verify the adapter attributes, run the following command:
# lsattr -El fscsi0
attach             switch          How this adapter is CONNECTED          False dyntrk             yes          Dynamic Tracking of FC Devices     True fc_err_recov        fast_fail   FC Fabric Event Error Recovery Policy True  scsi_id       0x10d00    Adapter SCSI ID                      False sw_fc_class       3        FC Class for Fabric controllers.    True

For
AIX 5100-06, after installing APAR IY48725, run the following commands:
# chdev -l fscsi0 -a fc_err_recov=fast_fail -P
Note: For FastFail to be fully enabled,  this attribute must be set on all adapters and controllers.

Storage Foundation for Oracle/RAC (SFRAC) 4.0 MP2 and Oracle 10.1.0.4 issues:
When installing Oracle 10.1.0.4 or upgrading to Oracle 10.1.0.4 with Storage Foundation for Oracle RAC 4.0 MP2,  some customers could experience problems. Symantec therefore recommends using SFRAC 4.0 MP3 if planning to install Oracle 10.1.0.4 or upgrade to Oracle 10.1.0.4.

Other issues:
The following issues have been identified after release. Contact Symantec Enterprise Technical Support for information on point patches for any of the issues if applicable
1. Minor issue encountered during the installation of the VRTSvmman package:
During installation of AxRT 4.0 MP2 of Storage Foundation, the VRTSvmman package will issue a message that it is already installed. This message is expected and can be ignored.

2. Read this when you upgrade your system to AIX 5.3 and are running 4.0MP2 release
  ############################################################################
Symantec has uncovered a critical problem with VERITAS File System on AIX 5.3. This issue affects all VERITAS Storage Foundation Products on AIX 5.3. Storage Foundation Products on AIX 5.1 or 5.2 are not affected.
Possible symptoms of this issue are failure to perform I/O to files or failure to start up an instance of Oracle or DB2 database.
The problem appears as soon the base operating system filesets of AIX, bos.mp or bos.mp64, are updated to the 5.3.0.31 level or higher. Installing certain APARs, such as IY69000, or updating AIX 5.3 to ML03 updates the base operating system fileset level to 5.3.0.31 or higher.
Symantec has issued a fix for this problem. Before installing the fix,  you must determine whether the Storage Foundation product in your AIX 5.3 environment requires the patch or not. To check this,  you can run the lslpp command, as shown below, and check the fileset level of the latest bos.mp or bos.mp64 installed on your system.
# lslpp -l bos.mp bos.mp64
If the displayed output shows the level below 5.3.0.31, you don't have to install the fix.
Storage Foundation products version 4.0 MP2 on AIX 5.3 are affected by this problem. Symantec has released a patch to resolve this problem. To determine if the patch is applicable to your Storage Foundation installation,  you must run the lslpp command, as shown below, to determine the version of VERITAS File System installed on your system:
# lslpp -l VRTSvxfs
For VRTSvxfs at 4.0.2.x level (for any x), the installed version of Storage Foundation is 4.0 MP2, and you must install VRTSvxfs.4.0.2.2-incr.bff.
Contact Symantec Enterprise Support to obtain the required patch

Related TechNote for AxRT 4.0 MP2:
Storage Foundation for Oracle/RAC: http://support.veritas.com/docs/278286  

4.0 MP3 Related Issues
Patches
For AIX 5.1
The minimum requirements are ML6 with APAR IY67577 and IY65263. The APARS are available for download and installation.
For AIX 5.2
The minimum requirements are ML4 with APARS IY67584, IY65717 and IY59620. The APARS are available for download and installation.
For AIX 5.3
The minimum requirements is ML3, currently no APARS.

Fuser Command Issue
The following fuser commands could cause a system crash:
fuser -d or fuser -cx
The following APARS will fix the problem.
IY74495 - AIX 5.1
IY72302 - AIX 5.2
IY72337 - AIX 5.3

I18N and Installer Issue
When using the product installer to install a VERITAS product on a remote server, and the remote server locale is set to Simplified Chinese (zh_CN), the installation will fail and display some Chinese error messages. These are three workarounds for this problem.
1.  Change all system locales to English before starting an installation and restore the locale to Simplified Chinese after the installation completes
2.  For an individual local system, run the "unset LANG" command before installation and uninstallation. Repeat for each remote system
3.  Install the VERITAS product locally on each host, then manually configure by running the "installer -configure" command

Storage Foundation for Oracle RAC in conjunction with Oracle 10gR2 support
This maintenance release introduces support for Storage Foundation for Oracle RAC to operate on Oracle 10gR2 databases. Symantec has completed internal quality testing and is currently running the formal Oracle certification test. Until formally certified, Symantec recommends using this release of Storage Foundation for Oracle RAC only in test and development environments, and not in a production environment. Check back with this TechNote for updates on the certification date.

Storage Foundation for Oracle/RAC with Cluster Volume Manager and VERITAS Volume Replicator

The following steps are needed to use Volume Replicator on Storage Foundation for Oracle RAC:
Packages: VRTSvrpro, VRTSvrw, and VRTSvrdoc have to be installed on the system before the upgrade to VERITAS Storage Foundation for Oracle RAC software to the MP3 level with Volume Replicator is enabled. Customers with an existing installation of VERITAS Storage Foundation for Oracle RAC 4.0 must install Volume Replication 4.0 packages from the Symantec FTP site <ftp://ftp.support.veritas.com> :
1. Downloadvvr.4.0.00.0.aix.tar.gz
2. Unzip and untar this tar file
3. Change the directory to /volume_replicator/pkgs
4. Install the Volume Replicator packages VRTSvrpro, VRTSvrw, and VRTSvrdoc
5. Insert the VERITAS Cluster File Solution 4.0 MP3 software disc
6. Run the installmp command to install maintenance patches for both Volume Replicator and Storage Foundation for Oracle/RAC
Refer to the VERITAS Storage Foundation for Oracle RAC Installation and Configuration Guide for information on upgrading

Availability of ASLs (Array Support Library)
ASLs are available for download from the Support site  http://support.veritas.com . Click on the Knowledge Base Search link, and select Volume Manager for UNIX from the Product pull-down menu. Enter the search phrase "VERITAS Enabled Arrays" (quotes must be included in the search). Any newly issued ASLs will be found at the same site.

The "installvvr -configure" command does not work correctly in VERITAS Storage Foundation Maintenance Pack 3.  If you need to reconfigure existing Volume Replicator settings, follow the instructions below:
Modifying the Volume Replicator tunables, ports, and log configuration parameters:
1. To modify the tunables, use either SMIT or the /usr/sbin/vxtune command to specify the required values
2. To modify the ports, use the /usr/sbin/vrport command to specify the ports that VVR should use
3. To modify the log configuration parameters, edit the /etc/vx/vras/vras_env file
Rebooting the system will start the Volume Replicator processes automatically. If you want to use Volume Replicator before the next system reboot, start the processes to enable replication by issuing the command /usr/sbin/vxstart_vvr
For more information, refer to the VERITAS Volume Replicator 4.0 Administrator's Guide

Storage Foundation for Oracle/RAC 4.0 MP3:
Storage Foundation for Oracle/RAC in conjunction with Oracle 10g R1/R2:
Symantec has discovered various issues with using Storage Foundation for Oracle/RAC 4.0 MP3 in an Oracle (10gR1/R2) environment. A patch has been released to address these issues. This patch is available for download from our ftp site. Keep in mind that this patch needs to go on top of 4.0 MP3 before performing any action. Also, this patch is only for Storage Foundation for Oracle/RAC 4.0 MP3 on AIX 5.2, or AIX 5.3.  Please ensure you are running one of these supported configurations before applying this patch.  Navigate to our ftp site for the patch:

External Pointers:
ftp:
ftp.veritas.com <ftp://ftp.veritas.com>
Login:  anonymous
Password:  email address
Dir:  /pub/products
Tar file:  4.0MP3+e509875.tar.gz
# cksum  4.0MP3+e509875.tar.gz
923506668    9088   4.0MP3+e509875.tar.gz
Directions for uncompressing and untarring:
# gunzip 4.0MP3+e509875.tar.gz
# tar xf 4.0MP3+e509875.tar
Note: Follow the instructions in the README

VERITAS Cluster Server 4.0 MP3:
'AllowNativeCliUser' was disabled as part of the security fix which went on top of VERITAS Cluster Server 4.0 MP2:
This fix also got rolled into Cluster Server 4.0 MP3, however, documentation for this fix didn't make it into the VERITAS Cluster Server 4.0 MP3 Release Notes. Nevertheless, a README file which accompanied the security fix was distributed to all affected customers. The README provides details of the change in behavior and the efficient way for Cluster Server customers to run HA commands.

From the README file which accompanies the security fix:
CHANGE IN BEHAVIOR
------------------
- Restrict Access to Cluster Nodes to Authorized VCS Users
You must set the value of the cluster attribute AllowNativeCliUsers to the default value of 0.  This attribute is no longer supported.
 - Use halogin command to Save Authentication for Running VCS Commands
When non-root users execute ha commands, they are prompted for their VCS user name and password to authenticate themselves. Use the halogin command to save the authentication information so that you do not have to enter your credentials every time you run a VCS command.
The command stores authentication information in the user's home directory. If you run the command for different hosts, VCS stores authentication information for each host.
 To log on to VCS:
 # halogin <vcsusername> <password>
To end a session for any host:
 # halogin -end session <hostname>
 To end a session for the local host
 # halogin -end session local host
To end all sessions:
 # halogin -endallsessions
 When you end all sessions, VCS prompts you for credentials every time you run a VCS command.

Dynamic Multipathing (DMP) Support of VSCSI LUNs
Storage Foundations 4.0 MP3 support for Advanced Power Virtualization (APV) of AIX 5.3 is limited to micropartitions of CPUs, virtual IP, and VSCSI LUNs. In a configuration where micropartitions have access to the same VSCSI LUN through multiple VIO servers, multipathing is currently supported by AIX MPIO only. In such scenarios DMP has access to only a single VSCSI path and hence multipathing functionality of DMP is disabled.

Storage Foundation for Oracle/RAC 4.0 MP2 and Oracle 10.1.0.4 issues:
When installing Oracle 10.1.0.4 or upgrading to Oracle 10.1.0.4 with Storage Foundation for Oracle RAC 4.0 MP2,  some customers could experience problems. It is therefore recommended to use Storage Foundation for Oracle/RAC 4.0 MP3 when planning to install Oracle 10.1.0.4 or upgrade to Oracle 10.1.0.4.

A fixed issue in 4.0 MP3 that wasn't documented in the 4.0 MP3 Release Notes
A cluster wide hang possibility exists during node leave reconfiguration in a Cluster Volume Manager (CVM) and Cluster File System (CFS) setup due to a CVM defect - Incident# e333917.
Problem description:
A leaving node triggers recovery operations in the cluster, as a result, CFS through CVM sends a PING message to all the nodes in the cluster and expects an acknowledgement from the member nodes. A cluster hang could potentially occur if CVM skips updating a bit in one of its message queues for the node that has left the cluster,  causing it to wait indefinitely in anticipation of a reply from the leaver node which is never going to come back. The hang is not expected to be hit in every node leave reconfiguration as it depends on the presence of a message in a particular message queue which can potentially happen if there are any failures sending a CVM message during the leaver node processing.
Workaround:
There is no known workaround. A reboot of all the cluster nodes is required to restore normalcy.
Resolution:
This VERITAS Volume Manager patch is included in the AxRT 4. 0 MP3 release.

Important issues pertaining to VERITAS Cluster Server (VCS) 4.0 MP2 and VCS 4.0 MP3 on AIX 5.3:
A jumbo patch for VCS networking agents on AIX was released in the first week of March 2006. This patch applies to VCS 4.0 MP2/MP3 running on AIX 5.3. This patch is available from Symantec Enterprise Technical Support.

Fixes and Enhancements Included in this VCS Patch ( VCS40MP3+JumboPatcha )
This patch includes the following fixes :
e494563   MultiNICB fails when device name length is
               greater than 3 characters. This fix was also
               included in VCS 4.0 MP3.
e542092   Clean entry point not implemented for IP
e529849   MultiNICB: IP Failover is incorrect when more
               than one VIP is configured.
e530312   NIC Agent does not report OFFLINE state on NIC failure
e520034   Support for multiple MultiNICA instances
e297779   Support for multiple MultiNICB instances
e519858   Report correct error message when haping fails
               because NetworkHost is on a different subnet

Known Issues and Workarounds
e565151
 LinkTest using 'haping -l' fails to correctly
              report NIC status.
The MultiNICB agent has an optional attribute called LinkTestRatio which controls the frequency of the pings the agent does to determine the status of an interface. If this attribute is set to 0, pings are not done at all and the link status of the driver is used to determine the state of the network interface. The default value is 1 which means that pings are done every monitor cycle. On AIX, the ethernet driver knows about link failure only from the software state but not from the hardware state (i.e. if you were to remove the back end cable into the network). If the LinkTestRatio is set to a value other than 1, the agent will rely on the OS driver telling it the status of the interface, and hence if the cable is pulled the agent will not detect a failure.
A forthcoming efix from IBM for PMR 87796,756 will address this issue. Until then make sure that LinkTestRatio is set to its default value of 1.
Note:  All other known issues and workarounds for VCS 4.0MP3 apply. Refer to the VCS 40 MP3 Release Notes for more information

Ether Channel Support
This patch was tested for Network Interface Backup (NIB) ether channel configuration with the IP/NIC agents. Per this configuration, there are for example two interfaces en1 and en2 connected to different switches. You can combine en1 and en2 to form an ether channel interface, say en3. VCS NIC/IP agents can then be used to monitor en3 and the virtual IP configured on it.

An issue pertaining to the VCS varyoffvg command in respect to the LVMVG resource:
Summary
e536453  On cable pull; failback of LVMVG resource fails ( fixed in 4.0MP3 )
In certain circumstances, the varyoffvg command does not deactivate all the volume groups on a node. This failure can prevent the failback of the LVMVG resource.

Details
In situations where storage connectivity is lost, the LVMVG resources fails over. Failback for the LVMVG resource requires the deactivation of the volume groups on the node that lost its connectivity to storage. VCS uses the varyoffvg command to deactivate the volume groups. The LVMVG resource cannot failback, however, when deactivation is unsuccessful.
When the volume group loses its storage connectivity, the clean script performs the varyoffvg command. Deactivation using the varyoffvg command can fail, however, if the volume group is busy. For example, when the volume group has pending I/O operations or when an application or an upper level resources in the resource dependency tree uses the volume group.
To overcome this deactivation failure, a post offline trigger has been added to issue the varyoffvg command. A side effect of the post offline trigger is that you must set the value of the OnlineRetryLimit attribute to 0.
After the restoration of storage connectivity, you must ensure that the volume groups are deactivated on the node. It is then possible to clear the fault on the resources. If you find active volume groups, deactivate them using the varyoffvg command.
The LVMVG resource must be the bottom most resource in the resource dependency tree in the service group. A resource below the LVMVG resource can potentially fail to go offline if the volume group's deactivation failed.

An issue pertaining to dynamic reconfiguration of processors on AIX:
Summary
e534744 VCS system panic after dynamic reconfiguration of processors on AIX
There is a potential for Veritas Cluster Server (VCS) to panic a system due to an issue with cluster heartbeat communications (LLT) on the AIX platform following a dynamic reconfiguration where processors are reallocated.  It has been observed that a dynamic increase in the number of processors may result in memory corruption caused by VCS heartbeat communications (LLT).  The actual instance of a VCS panic may not be immediate and may occur at a later time such as when VCS is active during failover or disaster recovery operations. The issue is addressed by VCS LLT patches for AxRT4.0 base, AxRT4.0MP2 and AxRT4.0MP3.  Point patches are available from Symantec Technical Support.

4.0 MP4 Related Issues:
System Requirements:
This release of Veritas Storage Foundation products requires the following minimum AIX operating system and the appropriate technology level (TL) and service pack (SP) versions:
AIX 5.2 TL8 and above
AIX 5.3 TL5 plus Service Pack 1 and above
Upgrade to AIX 5.2 or 5.3 and then install the appropriate TLs before installing Veritas Storage Foundation

No Longer Supported:
IBM has announced the withdrawal of support for AIX 5.1. Contact IBM customer support for more information.
Support for AIX 5.1 is not available in this release. Please upgrade your operating system before installing this maintenance pack.
Patches:
Recommended APAR:
IY91212  < http://www-1.ibm.com/support/docview.wss?uid=isg1IY91272>  on AIX 5.3 addresses an issue of a possible data anomaly exposure in AIX Fibre Channel Device Driver when Dynamic Tracking feature is enabled.  Since DMP requires that Dynamic Tracking enabled, installation of this APAR is strongly recommended.
Note: IY89283 addresses the above issue in AIX 5.2.

IBM DS4000 Storage Array Issues
:
- Upgrading a current configuration with an IBM DS4000 Storage Array to 4.0 MP4 requires the installation of a new ASL/APM. See Technote  http://support.veritas.com/docs/285540
- A problem has been identified in the 4.0 MP4 release related to the DS4000 disk array. This problem will cause an excessive delay in Cluster Volume Manager (CVM) node join time and will cause VCS resources to timeout and not come online during a system reboot. In addition, restarting vxconfigd generates numerous "reservation conflict" error messages in the error log which will temporarily impact system performance. Symantec has identified the root cause of the problem and is preparing a patch. This link will be updated when the patch is available.

Veritas Volume Manager issue:
Summary
e899127 vxdmpadm iostat command is not working on AIX
vxdmpadm iostat is not working with powerpath.
Powerpath 4.5.x  is supported with this production release of 4.0 MP4.  With the  powerpath software installed on a CX array, the vxmdmpadm iostat command fails to return valid data.
This is a sample command with the error returned:
vxdmpadm iostat show dmpnodename=EMC_CLARiiON0_10 interval=2 count=3 VxVM vxdmpadm ERROR V-5-1-8668 Invalid argument
All invocations of the vxdmpadm iostat command return this error message.

Daylight Saving Time Issues
For information about Daylight Saving Time issues, refer to this
technote:   http://support.veritas.com/docs/286461
 When you use the DB2 Setup wizard, db2setup, to install DB2 v9 product onto a VxFS file system directory, it will fail with the following error:
            DBI1501E an internal error was encountered.
            Explanation:
            An error was detected while performing an internal operation.
            User Response:
 Verify that the file system on which the file resides is not damaged. If the problem persists, contact IBM Support with the following information:
            1. Message number
            2. Internal error message description
            3. Problem description
  This is because DB2 Setup wizard is unable to recognize Storage Foundation VxVM volumes. To work around this problem, use the response file installation method for new installation.
 Work around procedure to install DB2 v9 product onto a VxFS file system directory using response file installation method:
1)      From the DB2 Setup launchpad, select install new to install the product.
2)      Then DB2 Setup Wizard will be launched. Accept the license agreement, and select the installation type.
3)      In the Select the installation, response file creation, or both window, select the Save my installation setting in a response file option. Then, in the

Response file name field, enter a response file name.
Notes:
a.       No software is installed on the computer, only the response file is created with the name specified in the Response file name field.
b.      If you select a partitioned database installation, two response files will be generated, one for the instance-owning computer and one for the participating computers. The participating computer response file name is generated based on the name of the instance-owning computer.
4)      In the Select the installation directory window, it contains a default directory. Do not try to enter VxFS file system directory here since it will fail. Leave the default directory as is and we will modify it later.
5)      Proceed through the rest of the installation panels selecting the options you want.
6)      When the installation has completed, the DB2 Setup wizard will have placed the generated response file in the path you specified.
7)      Edit the response file to change the installation directory. The installation directory is specified as, e.g.:
FILE      = /opt/IBM/db2/V9.1_01
Change the directory to an empty VxFS file system directory. Note that for partitioned database installation, two response files will be generated. Modify one or both of the response files, depends on where you want the DB2 product to install on each system.
8)      Install the DB2 product using the response file that you modified. Enter the db2setup command as follow:
<db2setup_directory>/db2setup -r <responsefile_directory>/<response_file>

Issue regarding the "Install Shield Multi Platform"(ISMP), an installer used by IBM WebSphere:

Problem Description:

This is a bug in the ISMP (install shield multi-platform), an installer used by the WebSphere. ISMP did not recognize the VxVM/VxFS to calculate the size of the volume.  ISMP fixed this in their 5.0.2 release. However, IBM did not pick this release up until WebSphere 6.0 release.
So, if customer is going to use WAS6.0, then it should install on VxFS. Prior versions continue to fail to install on VxFS. There is a  hotfix from ISMP that would make prior versions work as well.


The work-around is:

(1) copy the attached file to temp directory (Under the Related Documents section)
294985  ISMPfixforWAS5_294985.tar
(2) export AIX_LIB_LOC=/<absloute path to the file>

(3) run install again




Supplemental Materials

SourceiTools
Value144338
DescriptionCFS, stress got vx_iaccess:1a (vxfs defect 2)

SourceiTools
Value148646
DescriptionAdd "o" type (DS_VMZ) raw VxVM volume support for ODM (ODM Patch)

SourceETrack
Value222619
DescriptionVEA Incident for Out of Memory issue.

SourceiTools
Value148328
DescriptionAIX vx_vmm_buf_count too high on 32 bit kernel - can exhaust kern heap (Defect 1, vxfs patch)

SourceETrack
Value263556
Descriptionvxdctl may be slow with EMC Clariion CX arrays:

SourceETrack
Value333917
DescriptionA cluster wide hang possibility exists during node leave reconfiguration in a Cluster Volume Manager (CVM) and Cluster File System (CFS) setup due to a CVM defect

SourceETrack
Value494563
DescriptionMultiNICB fails when device name lengths is greater than 3 characters

SourceETrack
Value542092
DescriptionClean entry point not implemented for IP

SourceETrack
Value529849
DescriptionMultiNICB: IP failover is incorrect when more than one VIP is configured

SourceETrack
Value530312
DescriptionNIC agent does not report OFFLINE state on NIC failure

SourceETrack
Value520034
DescriptionSupport for multiple MultiNICB instances

SourceETrack
Value519858
DescriptionReport correct error message when haping fails because NetworkHost is on a different subnet

SourceETrack
Value565151
DescriptionLinkTest using 'haping -l fails to correctly report NIC status

SourceETrack
Value536453
DescriptionOn cable pull; failback of LVMVG resource fails.

SourceETrack
Value534744
DescriptionVCS system panic after Dynamic Reconfiguration of processors on AIX

SourceETrack
Value1061301
DescriptionDisabling/Enabling DMP path in special order would cause VxVM crashing and CFSmount fails.

SourceETrack
Value1065151
DescriptionDMP failover taking too long time after disabling 2 ports from array side.

SourceETrack
Value1057903
DescriptionDisk status change to 'Failing' or 'NODEVICE' when one path be disabled or reboot both nodes.

SourceETrack
Value865150
DescriptionCVM Reboot Resources Fault - (RETEST)Qualification AxRT SFOR 4.0mp4 DMP AIX 5.3 DS4300a0a


Legacy ID



269928


Article URL http://www.symantec.com/docs/TECH33563


Terms of use for this information are found in Legal Notices