Sybase program "startserver" failed to start the Sybase database server on a VCS node because of the extra long PATH environment varaible
| Article:TECH170330 | | | Created: 2011-09-26 | | | Updated: 2011-09-26 | | | Article URL http://www.symantec.com/docs/TECH170330 |
Problem
Sybase resource cannot be started on VCS server node1, but this resource can be onlined on the another server node2.
Sometime restarting HAD (hastop/hastart) on node1 caused the issue, but sometime the issue disappeared if HAD was restarted again.
Error
Sybase_A.log error messages:
-----------------------------
2011/09/02 13:16:06 VCS ERROR V-16-2-13066 Thread(3) Agent is calling clean for resource(sg1_syb_rs) because the resource is not up even after online completed.
2011/09/02 13:16:08 VCS ERROR V-16-2-13068 Thread(3) Resource(sg1_syb_rs) - clean completed successfully.
2011/09/02 13:16:08 VCS ERROR V-16-2-13071 Thread(3) Resource(sg1_syb_rs): reached OnlineRetryLimit(0).
-----------------------------
Sybase_A.log Debug information:
-------------------------------
2011/09/02 13:41:08 VCS DBG_1 V-16-50-0 Sybase:sg1_syb_rs:monitor:Cookie analysis finished. proc_ok is FALSE
Sybase.C:sybase_monitor[641]
2011/09/02 13:41:08 VCS DBG_1 V-16-50-0 Sybase:sg1_syb_rs:monitor:Could not find a process for data server SYB1. Returning OFFLINE.
Sybase.C:sybase_monitor[779]
2011/09/02 13:41:08 VCS DBG_AGINFO V-16-50-0 Thread(3) Resource(sg1_syb_rs) - monitor entry point exited with a confidence value 0.
VCSAgType.C:call_monitor[1409]
--------
2011/09/02 13:42:15 VCS DBG_AGDEBUG V-16-50-0 Thread(3) Calling online for resource sg1_syb_rs
VCSAgType.C:call_online[1563]
2011/09/02 13:42:46 VCS DBG_AGDEBUG V-16-50-0 Thread(3) script (/opt/VRTSagents/ha/bin/Sybase/online) exited with status (10)
VCSAgProcess.C:exec_script[1111]
2011/09/02 13:42:46 VCS DBG_AGINFO V-16-50-0 Thread(3) Resource(sg1_syb_rs) - online entry point exited with a return value 10.
VCSAgType.C:call_online[1594]
--------------------------------
---------------Use truss to check the SybaseAgent during online process ----------
root@osssvr-1 # ps -ef | grep -i sybase | grep -i agent
root 25991 1 0 Sep 08 ? 25:49 /opt/VRTSagents/ha/bin/Sybase/SybaseAgent -type Sybase -agdir /opt/VRTSagents/h
root 25993 1 0 Sep 08 ? 21:22 /opt/VRTSagents/ha/bin/SybaseBk/SybaseBkAgent -type SybaseBk -agdir /opt/VRTSag
root@osssvr-1 # truss -faile -o truss.out.0923 -p 25991
-------------------------------------------
The issue node truss info:
26134: 95.5537 sigaction(SIGINT, 0xFFFFFFFF7FFFCCC0, 0x00000000) = 0
26134: 95.5538 sigaction(SIGQUIT, 0xFFFFFFFF7FFFCCC0, 0x00000000) = 0
26132: 95.5544 lwp_sigmask(SIG_SETMASK, 0x00020000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
26134: 95.5550 execve("/bin/sh", 0xFFFFFFFF7FFFCEC8, 0xFFFFFFFF7FFFE1F8) argc = 3
26134: argv: sh -c bin:/usr/sbin:/sbin <<<<<<<<<<< Sybase startserver tried to run a non-existent program and caused the online failure
The truss output from a successful Sybase resource online:
28388: 92.1887 lwp_sigmask(SIG_SETMASK, 0x00020000, 0x00000000) = 0xFFBFFEFF [0x0000FFFF]
28390: 92.1893 execve("/bin/sh", 0xFFFFFFFF7FFFD2D8, 0xFFFFFFFF7FFFE608) argc = 3
28390: argv: sh -c /export/home/sybase/ASE-15_0/install/RUN_SYB1 & <<<<<<<<< Sybase startserver should execute this program
--------------------------------------------------------------------
Environment
------- Customer has an extra long PATH environment variable ---------------------------------
Use the "pargs" command to check the process environment variables.
# pargs -e 25991
25991: /opt/VRTSagents/ha/bin/Sybase/SybaseAgent -type Sybase -agdir /opt/VRTSagents/ha
envp[0]: CLUSTER_LOGDBG=DBG_1 DBG_2 DBG_3 DBG_4 DBG_5 DBG_AGINFO DBG_AGDEBUG
.....
.....
envp[63]: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/sbin:/opt/VRTSvcs/bin:/opt/VRTS/bin:
/export/home/omc_control/osswebtools/web3/iManagerM2000V200R011C00SPC210/tools/python:/opt/OMC/3rdTools/bin:
/opt/OMC/3rdTools/python/bin:/opt/OMC/jre/bin:/opt/OMC:/opt/OMC/bin:/opt/OMC/3rdTools:/usr/local/bin:/opt/SUNWspro/bin:.:
/export/forte6/SUNWspro/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:.:/export/forte6/SUNWspro/bin:/opt/OMC/3rdTools/postgresql/bin:/opt/OMC/3rdTools/usr/local/bin:/opt/OMC/bin:/opt/OMC/3rdTools/bin:/opt/OMC/3rdTools/flex-2.5.4/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:/opt/OMC/3rdTools/sybase/OCS-15_0/bin:/opt/sybase/ASE-15_0/jobscheduler/bin:/opt/sybase/RPL-15_0/bin:/opt/sybase/ASEP/bin:/opt/sybase/DBISQL/bin:/opt/sybase/UAF-2_0/bin:/opt/sybase/OCS-15_0/bin:/opt/sybase/ASE-15_0/bin:
/opt/sybase/ASE-15_0/install:/opt/OSMU/omc_control/bin:/opt/OSMU/omc_control/lib/python:/usr/sbin:/sbin:/opt/VRTSvcs/bin:/opt/VRTS/bin:/usr/sbin:/sbin:
/opt/VRTSvcs/bin:/opt/VRTS/bin:/usr/sbin:/usr/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,SPARC-Enterprise-T5220/sbin:
/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin:
/usr/openv/netbackup/bin:/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/lib/vxvm/bin:/opt/VRTS/bin:
/opt/VRTS/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,SPARC-Enterprise-T5220/sbin:
/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin:/usr/openv/netbackup/bin:
/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/lib/vxvm/bin:/opt/VRTS/bin:/opt/VRTS/bin:/usr/dt/bin:/sbin:/usr/sbin:
/usr/sfw/bin::/opt/OMC/3rdTools/sms/script:/usr/ibmcxx/bin:/opt/OMC/3rdTools/sms/bin::/opt/OMC/3rdTools/sms/script:/usr/ibmcxx/bin:/opt/OMC/3rdTools/sms/bin:
/usr/sbin:/sbin:/usr/ccs/bin:/usr/openwin/bin:/usr/dt/bin:/usr/platform/SUNW,SPARC-Enterprise-T5220/sbin:
/opt/sun/bin:/etc/vx/bin:/opt/VRTS/bin:/etc/vx/bin:/opt/VRTSob/bin:/opt/VRTSvlic/bin:/opt/SUNWexplo/bin:/opt/SUNWsneep/bin:/opt/CTEact/bin:/usr/openv/netbackup/bin:
/usr/openv/netbackup/bin/admincmd:/usr/openv/netbackup/bin/goodies:/usr/openv/volmgr/bin:/usr/lib/vxvm/bin:/opt/VRTS/bin:/opt/VRTS/bin
.....
.....
---------------------------------------------------------------------------------
Cause
The HAD and Sybase Agent processes's extra long PATH environment variable was inherited from the user shell where VCS daemon was started. (hastart)
The user performed some maintenance operations and changed the PATH variable during the time and then restarted VCS. The changed PATH variable was inherited by the VCS/Agent processes. Sybase failed to online when the extra long PATH variable was inherited by the Sybase startserver program through the VCS Sybase Agent process.
Below testing shows how the user shell PATH environment can be inherited by the VCS SybaseAgent process.
---------------------------------------------------------------------------------
The environment variables are inherited by all the related VCS processes when VCS daemon (HAD) is restared. This includes the VCS Sybase agent process.
If the system boots up afresh, then the processes will have clean environment variables.
------------------------------------------------------
# echo $PATH <<<<<<<user shell PATH variable which is set to extra long
/usr/sbin:/usr/bin:/etc/vx/bin:/usr/share/webconsole/bin:/sbin:/usr/lib/fs/vxfs/bin:/usr/lib/vxvm/bin:/usr/lib/vxvm/voladm.d/bin:
/usr/openwin/bin:/usr/perl5/bin:/opt/VRTS/bin:/opt/VRTSvlic/bin:/opt/VRTSperl/bin:/opt/VRTSob/bin:/opt/VRTSob/jre/bin:
/opt/VRTSvxvm/vmcvt/bin:/opt/VRTSfspro/bin:/opt/VRTSalloc/bin:/opt/VRTSvcs/vxfen/bin:/opt/VRTSvcs/bin:
/opt/VRTSvcs/wizards/bin:/opt/VRTSweb/bin:/opt/VRTScssim/bin:/opt/VRTSvail/bin:/opt/VRTSdb2ed/bin:/etc/vx/diag.d:
/opt/VRTSspt/FS/MetaSave:/opt/VRTS/bin:/opt/VRTSspt/FS/VxBench
# pargs -e 4062
4062: /opt/VRTSvcs/bin/had <<<<<<had process
envp[0]: EAT_CATALOG_DIR=/opt/VRTSvcs
envp[1]: EAT_HOME_DIR=/opt/VRTSvcs
envp[2]: PWD=/var/VRTSvcs/diag/had
envp[3]: VCS_DIAG=/var/VRTSvcs
envp[4]: VCS_LOG=/var/VRTSvcs
envp[5]: VCS_CONF=/etc/VRTSvcs
envp[6]: VCS_HOME=/opt/VRTSvcs
envp[7]: LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
envp[8]: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/opt/VRTSvcs/bin <<<<<<<the clear PATH variable
envp[9]: SMF_FMRI=svc:/system/vcs:default
envp[10]: SMF_METHOD=/lib/svc/method/vcs start
envp[11]: SMF_RESTARTER=svc:/system/svc/restarter:default
envp[12]: TZ=PRC
# hastop -local –force <<<<<<restart HAD on this session
# hastart
# pargs -e 9617 <<<<<check the HAD process again after restarted.
9617: /opt/VRTSvcs/bin/had
envp[0]: EAT_CATALOG_DIR=/opt/VRTSvcs
envp[1]: EAT_HOME_DIR=/opt/VRTSvcs
envp[2]: VCS_DIAG=/var/VRTSvcs
envp[3]: VCS_LOG=/var/VRTSvcs
envp[4]: VCS_CONF=/etc/VRTSvcs
envp[5]: VCS_HOME=/opt/VRTSvcs
envp[6]: HOME=/
envp[7]: HZ=100
envp[8]: LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
envp[9]: LOGNAME=root
envp[10]: MAIL=/var/mail/root
envp[11]: MANPATH=:/usr/share/man:/opt/VRTS/man:/opt/VRTSvlic/man:/opt/VRTSob/jre/man:/opt/VRTSdb2ed/man:/opt/VRTSd2gui/man
envp[12]: PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/sbin:/usr/bin:/etc/vx/bin:/usr/share/webconsole/bin:/sbin:/usr/lib/fs/vxfs/bin:/usr/lib/vxvm/bin:
/usr/lib/vxvm/voladm.d/bin:/usr/openwin/bin:/usr/perl5/bin:/opt/VRTS/bin:/opt/VRTSvlic/bin:/opt/VRTSperl/bin:/opt/VRTSob/bin:
/opt/VRTSob/jre/bin:/opt/VRTSvxvm/vmcvt/bin:/opt/VRTSfspro/bin:/opt/VRTSalloc/bin:/opt/VRTSvcs/vxfen/bin:/opt/VRTSvcs/bin:/opt/VRTSvcs/wizards/bin:
/opt/VRTSweb/bin:/opt/VRTScssim/bin:/opt/VRTSvail/bin:/opt/VRTSdb2ed/bin:/etc/vx/diag.d:/opt/VRTSspt/FS/MetaSave:
/opt/VRTS/bin:/opt/VRTSspt/FS/VxBench <<<<The PATH is inherited from user shell.
----------------------------------------------------------------------------
Below testing showed the Sybase "startserver" having issue if PATH variable was too long.
If the PATH variable is more then 2045 characters, then Sybase startserver program will fail to start the Sybase database server.
---------------------------------------------------------------------
The issue is reproduced in the following way. The Sybase "startserver" fails to start the Sybase database server if the PATH variable is too long.
# su dbuser
$ . /export/home/sybase/SYBASE.sh
$ PATH=$PATH:very_long_path_variable:...... # set the PATH environment variable to have more than 2048 characters
$ PATH=$PATH:very_long_path_variable:...... # set the PATH environment variable to have more than 2048 characters
$ echo $PATH
$ echo $PATH | wc
$ /export/home/sybase/ASE-15_0/install/startserver -f /export/home/sybase/ASE-15_0/install/RUN_SYB1
---------------------------------------------------------------------
Solution
If the PATH variable is added to /opt/VRTSvcs/bin/vcsenv, the HAD/Agent processes will inherit the PATH variable from "/opt/VRTSvcs/bin/vcsenv", not from the user shell and the problem can be fixed.
Below testing shows that the PATH environment will be inherited from "/opt/VRTSvcs/bin/vcsenv" if PATH environment variable is defined in it.
------------------------------------------------------------------------------------
# echo “PATH=/usr/sbin:/usr/bin:/etc/vx/bin:/opt/VRTS/bin:/opt/VRTSvcs/bin:/test/test” >> /opt/VRTSvcs/bin/vcsenv
# echo $PATH
/usr/sbin:/usr/bin:/etc/vx/bin:/usr/share/webconsole/bin:/sbin:/usr/lib/fs/vxfs/bin:/usr/lib/vxvm/bin:/usr/lib/vxvm/voladm.d/bin:/usr/openwin/bin:
/usr/perl5/bin:/opt/VRTS/bin:/opt/VRTSvlic/bin:/opt/VRTSperl/bin:/opt/VRTSob/bin:/opt/VRTSob/jre/bin:/opt/VRTSvxvm/vmcvt/bin:/opt/VRTSfspro/bin:
/opt/VRTSalloc/bin:/opt/VRTSvcs/vxfen/bin:/opt/VRTSvcs/bin:/opt/VRTSvcs/wizards/bin:/opt/VRTSweb/bin:/opt/VRTScssim/bin:/opt/VRTSvail/bin:
/opt/VRTSdb2ed/bin:/etc/vx/diag.d:/opt/VRTSspt/FS/MetaSave:/opt/VRTS/bin:/opt/VRTSspt/FS/VxBench:/test/test
# hastop -local -force
# hastart
# ps -ef | grep had
root 12386 11496 0 11:08:11 pts/1 0:00 grep had
root 12378 1 0 11:08:07 ? 0:00 /opt/VRTSvcs/bin/had
root 12380 1 0 11:08:07 ? 0:00 /opt/VRTSvcs/bin/hashadow
# pargs -e 12378
12378: /opt/VRTSvcs/bin/had
envp[0]: EAT_CATALOG_DIR=/opt/VRTSvcs
envp[1]: EAT_HOME_DIR=/opt/VRTSvcs
envp[2]: VCS_DIAG=/var/VRTSvcs
envp[3]: VCS_LOG=/var/VRTSvcs
envp[4]: VCS_CONF=/etc/VRTSvcs
envp[5]: VCS_HOME=/opt/VRTSvcs
envp[6]: HOME=/
envp[7]: HZ=100
envp[8]: LD_LIBRARY_PATH=/opt/VRTSvcs/lib:
envp[9]: LOGNAME=root
envp[10]: MAIL=/var/mail/root
envp[11]: MANPATH=:/usr/share/man:/opt/VRTS/man:/opt/VRTSvlic/man:/opt/VRTSob/jre/man:/opt/VRTSdb2ed/man:/opt/VRTSd2gui/man
envp[12]: OLDPWD=/
envp[13]: PATH=/usr/sbin:/usr/bin:/etc/vx/bin:/opt/VRTS/bin:/opt/VRTSvcs/bin:/test/test <<<<<<The PATH variable from "/opt/VRTSvcs/bin/vcsenv"
---------------------------------------------------------------------------------------
|
|
Article URL http://www.symantec.com/docs/TECH170330
Terms of use for this information are found in Legal Notices
Thank you.