Veritas Storage Foundation (tm) for Oracle RAC 5.0 for HP-UX 11iv2 Patch: 5.0MP1CP1 - PVKL_03769

Article:TECH53011  |  Created: 2007-01-06  |  Updated: 2008-01-17  |  Article URL http://www.symantec.com/docs/TECH53011
Article Type
Technical Solution


Environment

Issue



Veritas Storage Foundation (tm) for Oracle RAC 5.0 for HP-UX 11iv2 Patch: 5.0MP1CP1 - PVKL_03769

Solution




After clicking on the Download Now link below, move the 5.0MP1CP1.tar_290461.gz file to a new directory and use these commands to unpack the file:

# mv 5.0MP1CP1.tar_290461.gz 5.0MP1CP1.tar.gz

# cksum 5.0MP1CP1.tar.gz
346114825       17582749        5.0MP1CP1.tar.gz

# gzip -d 5.0MP1CP1.tar.gz

# tar -xf 5.0MP1CP1.tar

# ls

5.0MP1CP1

# cd 5.0MP1CP1

# ls

copyright          patches            README_PVKL_03769


README_PVKL_03769 is Incomplete

The patch Installation instructions included in the README_PVKL_03769 file are incomplete.

Use the patch instructions included below.

Patch Name:: PVKL_03769

Creation Date:: 07/10/2007

Hardware Platforms - OS Releases::
s700: 11.23
s900: 11.23

Patch Description::

s700_800 11.23 VRTSdbac 5.0 MP1 Cumulative Patch 1

Symptoms::

PVKL_03769:

(SYMANTEC Incident Number:1068838)
In SFRAC environment, the cssd (Cluster Resource
Services Daemon) resource reports incorrect state.

(SYMANTEC Incident Number:1068827)
For supporting multiple Oracle 10g database
instances, main.cf file needed to be enhanced.

(SYMANTEC Incident Number:1069682)
Execution of following commands to configure
vcsmm (Veritas Cluster Server Multiplexing Module)
results in a system panic due to null pointer
dereference: -

# vcsmmconfig -c &
# vcsmmde

The stack trace consists of the following kernel
functions:

unix:trap()
vcsmm:vrfsm_gab_recv()
gab:gab_deliver_loop()
gab:gab_receive()
gab:gab_receive_port_que()
gab:gab_receive_que()
gab:gab_lrecv()
llt:llt_lrsrv_port()
llt:llt_deliver()

(SYMANTEC Incident Number:965232)
On HP-UX platform, the LMX helper thread is enabled
by default, even though it is intended to be
disabled by default.

(SYMANTEC Incident Number:1053546)
In SFRAC environment, with Oracle 10g R2 (10.2.0.3),
if the file $CRS_HOME/log/$HOST_NAME/cssd/
$HOST_NAME.pid is removed manually, an attempt to
online the cssd (Cluster Resource Services Daemon)
resource succeeds however, the 'hares -state'
command reports the state as OFFLINE|STATE UNKNOWN.

(SYMANTEC Incident Number:1069631)
On a VCS cluster running SFRAC, when one of the
nodes crashed, VCS reformed on the remaining nodes.
However, the Oracle RAC instance running on the
remaining node may crash with the Oracle alert log
emitting messages as follows:

ORA-27508: IPC error sending a message
ORA-27300: OS system dependent operation:dsaw: IPC
wire 0x8000000100318b9 failed with status: 0  
ORA-27301: OS failure message: Error 0  
ORA-27302: failure occurred at: vcsipc_dosnd  

The Oracle LMON trace file would contain error
messages as follows:

*** ERROR: INTERNAL: VCSIPC 05-03 22:56:20,
ipcmain.c:5636, c0x80000001001b4680
dsaw: IPC wire 0x8000000100318b90 present,
LMX node status OPEN,PNRD: Destroying wire:
nd 1 OPEN,PNRD"

(SYMANTEC Incident Number:1069656)
Oracle error ORA-0600 is displayed for Oracle
versions supporting SKGXP versions starting from
SKGXP24, when one of the nodes is viewed by LMX as
disconnected and Oracle issues a skgxpdis () on
the connection handles associated with that wire.

(SYMANTEC Incident Number:1068824)
After upgrading from oracle 10.2.0.2 to oracle
10.2.0.3, linkrac command fails to link Oracle
with VCSMM library, emitting an error message
"ERROR: Oracle binary not linked with Veritas
VCSMM library".

(SYMANTEC Incident Number:1068819)
linkrac command fails to link Oracle with VCSMM
library, emitting an error message
"/opt/VRTSvcs/rac/bin/oramake[100]: test: A ]
character is missing."

(SYMANTEC Incident Number:1069665)
The VCS IPC version is shown as 'TEST' in the
Oracle Alert Log.

(SYMANTEC Incident Number:1068813)
On a VCS cluster, an attempt to do private IP
failover using 'lltconfig' fails on the node where
the command is executed even though it succeeds on
other nodes of the cluster.

Defect Description::

PVKL_03769:

(SYMANTEC Incident Number:1068838)
The cssd-monitor script incorrectly used $HOSTNAME
instead of `$HOSTNAME` wherein HOSTNAME was set to
/bin/hostname, to extract the hostname from the
system. While filtering the output of 'hostname'
command, there will be a script error that results
in monitor script indicating incorrect state to CRS.

Resolution:
cssd-monitor script is modified to use `$HOSTNAME`
in all places instead of $HOSTNAME.

(SYMANTEC Incident Number:1068827)
The main.cf configuration file was to be updated
to support multiple Oracle database instances.

Resolution:
The main.cf file has been updated with
configuration syntax for support of multiple
database instances.

(SYMANTEC Incident Number:1069682)
During the init of vcsmm module, the vcsmm global
structure is initialized by calling
vcsmm_init_struct(). The latter function attempts
to log a debug message using VCSMM_DBG API. However,
the initialization of vcsmm global debug log was
done later by calling vcsmm_init_deblog(). This
resulted in accessing of un-initialized deblog
structure, leading to null pointer dereference and
hence the panic.

Resolution:
In the vcsmm_init() function, the call to
vcsmm_init_deblog() was made prior to the call to
vcsmm_init_struct() to initialize the structures
in correct order.

(SYMANTEC Incident Number:965232)
The tunable 'lmx_update_enabled' is set using
kctune API, which is expected to obtain the
LMX_UPDATE_DFL value of LMX_UPDATE_DISABLED.
However, lmx.modmeta file still had the value of
1 for default value of lmx_update_enabled.
This led to the helper thread being enabled by
default.

Resolution:
The LMX modmeta file was modified to set default
value of tunable lmx_update_enabled to 0.

(SYMANTEC Incident Number:1053546)
Until 10g R2, the pid of ossd was kept by CRS in
a file located at $CRS_HOME/log/$HOST/cssd/
$HOST.pid. Accordingly, the cssd-monitor script
used to do a health check of cssd based on whether
it found the pid or not. However, in 10g R3, no
such pid file is created by CRS, which resulted
in incorrect state being reported by VCS.

Resolution:
If the pid file is not seen in the traditional
location, cssd monitor script attempts to get
the pid from system process list by issuing a
'ps' command with appropriate options.

(SYMANTEC Incident Number:1069631)
When Oracle does a vectored send of a message
through skgxpvsnd(), the LMX IPC mechanism would
pass the messages across the wire and then does a
poll through LMX.  In certain cases if the
connection is down, LMX will set port connection
status to LMX_POLLSEND_NOTCONN to indicate that
the remote node is not connected. LMX IPC calls
vcsipc_destwr(), which in turn would mark all the
request handles as failed and calls skgxpdis().
Oracle has clarified that it does not expect
failed request handles on a skgxpvsnd() call
and would react catastrophically to ECONABORTED.

Resolution:
LMX IPC is modified to ignore LMX_POLLSEND_NOTCONN
if it is received on a vectored send, and mask
ECONABORTED. In this scenario LMX IPC will not
call vcsipc_destwr() and just pretend as though
nothing happened and continue. The Membership
Module (VCSMM) will detect the change in node
membership and do appropriate cleanup of requests.

(SYMANTEC Incident Number:1069656)
Stale request handles may be left behind by LMX
in the context.s doneq after the connection is
closed by calling skgxpdis(). VCS IPC will issue
skgxpdis () on all connections associated with
the wire, when it sees LMX_POLLSEND_NOTCONN as
the node status for one of the node members.
However, the request handles for the connection
are put on the context.s doneq, with their
connection state indicating it as CLOSED and
connection handle is freed. Such stale request
handles will result in ORA-0600 if Oracle
version SKGXP24 and above issues a call to
skgxpdis() and this in turn leads to accessing
of freed request handles.

Resolution:
For SKGXP version 2.4 and above, for any failed
/completed asynchronous requests on a
disconnected wire, if skgxpdis() is called by
Oracle, then after the completion of skgxpdis()
the request handles for the connections are not
queued in the context.s doneq. However, if VCS
IPC calls skgxpdis,() then VCS IPC will add the
request handles corresponding to the connection
handle, to the doneq but will leave the
connection handle as it is.  The request handles
on the doneq that are associated with this
connection handle will be deleted when Oracle
calls skgxpdis().

(SYMANTEC Incident Number:1068824)
The linkrac script functionality is broken
because, starting from version 10.2.0.3, Oracle has
uses the library "libskgxn2.a" instead of
"libskgxn2.so".

Resolution:
The linkrac script invokes /opt/VRTSvcs/rac/
bin/oramake utility to link the Oracle binary
with Veritas library. The issue is resolved by
moving the LIBSKGXN library at $ORACLE_HOME/
rdbms/lib/libskgxn2.a as ${ORACLE_HOME}/rdbms/
lib/$ libskgxn2.oracle.a in the 'oramake' script
before linking and restoring it back after the
linking is complete.

(SYMANTEC Incident Number:1068819)
The linkrac script invokes /opt/VRTSvcs/rac/bin/
oramake utility to link the Oracle binary with
Veritas library. There were script syntax errors
due to a space preceding the "]"

Resolution:
The issue is resolved by removing the  space
before the "]" in the test statements.

(SYMANTEC Incident Number:1069665)
Due to a parallel build started from vcsops/
local/scripts/buildall script that is used to
build the VCS IPC library, the IPC version file
gets over-written by a different build process
of different skgxp[xx] version. This results in
inconsistent IPC version, with the value shown
as 'TEST' in the Oracle Alert Log.

Resolution:
The buildipc script is modified to create
IPCVERSION file in specific skgxp[xx] directory
thereby avoiding over-writing of the IPC version.

(SYMANTEC Incident Number:1068813)
When the link on which IP was configured is not
a LLT link, the PrivNIC agent that is responsible
for failover of Private IP, erroneously decides
that there is no need to failover.

Resolution:
PrivNIC agent is modified to failover Private IP,
if the link on which the IP was configured is not
a LLT link.

Enhancement:: No

Products:: VRTSdbac 5.0

Product Directory::  /

Automatic Reboot?:: No

Status:: GR (General Release)

Filesets::
      PVKL_03769.VCSMM-KRN-64,r=1.0,a=HP-UX_B.11.23_IA/PA,v=VERITAS,fr=1.0,fa=HP-UX_B.11.23_PA
      PVKL_03769.VCSMM-KRN-64,r=1.0,a=HP-UX_B.11.23_IA/PA,v=VERITAS,fr=1.0,fa=HP-UX_B.11.23_IA

what(1) output::

cksum(1) Output::

Patch Conflicts:: None

Patch Dependencies::
      s700: 11.23: PVKL_03688 (5.0 MP1)
      s800: 11.23: PVKL_03688 (5.0 MP1)

Hardware Dependencies:: None

Other Dependencies:: None

Critical:: Yes

OS Kernel Files?:: Yes

Encrypt Files?::
No
SD Configure File?::
No
SD Unconfigure File?::
No
SD Verify File?::
No
SD Checkinstall File?::
No
SD Checkremove File?::
No
SD Preinstall File?::
No
SD Postinstall File?::
No
SD Preremove File?::
No
SD Postremove File?::
No

Release Notes::

Before Getting Started:

This patch is for SFRAC 5.0 MP1 on HP-UX. You must
upgrade to SFRAC 5.0 MP1 before applying the patch.
Refer to the "VERITAS Storage Foundation for Oracle
RAC 5.0 MP1 Installation and Configuration Guide"
for installing SFRAC 5.0 and SFRAC 5.0 MP1.

NOTE: Removing 5.0MP1 CP1 will take the system back
to 5.0MP1.


New In This Patch:

Oracle 10.2.0.3 support is added for SFRAC 5.0MP1.

Patch Installation Instructions:

Please follow below steps for installing PVKL_03769
on SFRAC 5.0 MP1 Cluster.

1. Stop VCS on all nodes
      # hastop -all

      Note: If Oracle Database & Clusterware is not under VCS Control, you need to stop database & clusterware manually before stopping VCS.


2. Unconfigure vcsmm.
      # vcsmmconfig -U

3. Run the following commands on each of the cluster
   nodes:
      # cd <patch_location>
      # swinstall -s `pwd` -x autoreboot=true PVKL_03769

4. After the installation is complete, verify that the patch
   has been installed.

      # swlist PVKL_03769

  If the patch installation is successful, the output
   resembles:
      
      # PVKL_03769            1.0            VRTS 5.0 MP1CP1 VRTSdbac Patch
        PVKL_03769.DBAC-KRN   1.0            Veritas LMX and VCSMM kernel files
        PVKL_03769.DBAC-RUN   1.0            Veritas DBAC generic files
        PVKL_03769.LMX-RUN    1.0            Veritas LMX userland files

      # kcmodule -v vcsmm

  If the patch installation is successful, it must have revision
   1.0:

      Module              vcsmm  (1.500.1)
      Description         VERITAS Membership Manager revision 5.0.1.0
      State               loaded (as requested)
      State at Next Boot  loaded (as requested)
      Capable             loaded auto unused
      Depends On          module gab:0.0.0
                          interface HPUX_11_23:1.0

5. Starting VCSMM
  Run the following command on each cluster node to start VCSMM
      # /sbin/init.d/vcsmm start

6. Take the Oracle resources offline

On each node, take the Oracle resources in the VCS configuration file
(main.cf) offline.

# hagrp -offline oracle_group -sys node_name

For example:

# /opt/VRTSvcs/bin/hagrp -offline Oracle1 -sys galaxy # /opt/VRTSvcs/bin/hagrp -offline Oracle1 -sys nebula

These commands stop the Oracle resources under VCS control running on the specified nodes.

7. Relink the SF Oracle RAC libraries to Oracle For 9i, please refer to " Veritas Storage Foundation� for Oracle RAC 5.0 MP1 Installation and Configuration Guide", "Relinking the SF Oracle RAC libraries to Oracle9i"
section in Chapter 7:" Installing Oracle9i RAC".

For 10g, please refer to "Veritas Storage Foundation� for Oracle RAC 5.0 MP1 Installation and Configuration Guide", "Relinking the SF Oracle RAC libraries to Oracle 10g" section in Chapter 13: "Installing Oracle 10g RAC".


Patch Removal Instructions:

Please follow below instructions for removing PVKL_03769:

1. Stop VCS on all nodes.
   
    # hastop -all

    Note: If Oracle Database & Clusterware is not under VCS Control, you need to stop database & clusterware manually before stopping VCS.

2. Remove the patch on each system using the swremove command:
      # swremove -x autoreboot=true PVKL_03769

3. Verify that the patch has been removed from each system. Type:
      # swlist PVKL_03769

  If the patch removal is successful,
      ERROR:   Software "PVKL_03769" was not found on host

Other Software Installation Instructions:

Installing ORACLE 10gR2:

Please refer to the "VERITAS Storage Foundation for Oracle
RAC 5.0 MP1 Installation and Configuration Guide", Chapter 5:
"Installing Oracle 10g Software" for installation of Oracle
10gR2 with the following exception:

* In the section "Installing Oracle 10g CRS," in step 7 (c, d),
  use init.cssd-10gR2.patch instead of init.cssd.patch as follows:

a. Change to the directory where the patch is to be copied:
      # cd /oracrs/css/admin
      # cp /opt/VRTSvcs/rac/patch/init.cssd-10gR2.patch .

b. Run the following command to install the patch:
      # patch < init.cssd-10gR2.patch init.cssd

Upgrading and migrating ORACLE Software:

Please follow the upgrade paths documented in "VERITAS Storage
Foundation for Oracle RAC 5.0 MP1 Release Notes" to upgrade
to SFRAC 5.0 MP1.

Instead of the step "Upgrading and Migrating Oracle Software,"
do the following:

1. Install the SFRAC 5.10 MP1 CP1. Follow the "INSTALLING THE
   SFRAC 5.0 MP1 CP1 ON SFRAC 5.0 MP1 CLUSTER" instructions.

2. Install Oracle 10gR2. Follow the "INSTALLING ORACLE 10gR2"
   instructions.

3. Apply Oracle Patchsets. Follow the "APPLYING ORACLE PATCHSETS"
   instructions.

4. Upgrade the database to Oracle 10gR2. Refer to the "Oracle
   Database Upgrade Guide 10g Release 2 (10.2) Part Number
   B14238-01."

Applying ORACLE Patchsets:

Please refer to the "VERITAS Storage Foundation for Oracle RAC
5.0 MP1 Installation and Configuration Guide", "Applying Oracle
Patchset" section in Chapter 6: "Upgrading and Migrating Oracle
Software".

In step 2, when you apply patchset to Oracle CRS, make sure to
patch init.cssd before you run root<xxx>.sh script as follows:

a. Change to the directory to which you want to copy the patch:
      # cd /oracrs/install/patch<xxx>/css/admin
      # cp /opt/VRTSvcs/rac/patch/init.cssd-10gR2.patch .

b. Run the following command to install the patch:
      # patch < init.cssd-10gR2.patch init.cssd

Known Issues: None



Attachments

5.0MP1CP1.tar_290461.gz (17.2 MBytes)


Legacy ID



290461


Article URL http://www.symantec.com/docs/TECH53011


Terms of use for this information are found in Legal Notices