Which version of VCS for HPUX11v2 fixes the error "VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group"?

Article:TECH161118  |  Created: 2011-05-27  |  Updated: 2012-07-21  |  Article URL http://www.symantec.com/docs/TECH161118
NOTE: If you are experiencing this particular known issue, we recommend that you Subscribe to receive email notification each time this article is updated. Subscribers will be the first to learn about any releases, status changes, workarounds or decisions made.
Article Type
Technical Solution


Environment

Issue



VCS issues the errors "VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group" when the maxfiles on the system runs out of.


Error



 

[ ERROR MESSAGES ]
1. /var/adm/OLDsyslog.log on the node, SYMC-HPUX1
---------------------------------------------------------------------------
May 25 12:27:19 SYMC-HPUX1vmunix: file: table is full
May 25 12:27:23 SYMC-HPUX1MQSeries: FFST record created in /var/mqm/errors/AMQ816.0.FDC
May 25 12:27:23 SYMC-HPUX1MQSeries: FFST (23,22,1,536909158) failed: /var/mqm/errors/AMQ16050.0.FDC (errno=23)
May 25 12:27:23 SYMC-HPUX1MQSeries: FFST (23,402,11,536895769) failed: /var/mqm/errors/AMQ16050.0.FDC (errno=23)
May 25 12:27:23 SYMC-HPUX1MQSeries: FFST (23,402,11,536895769) failed: /var/mqm/errors/AMQ16050.0.FDC (errno=23)
May 25 12:27:24 SYMC-HPUX1 above message repeats 3 times
---------------------------------------------------------------------------
 
2. /var/VRTSvcs/log/Application_A.log on the node, SYMC-HPUX1
---------------------------------------------------------------------------
2011/05/25 12:27:20 VCS WARNING V-16-10021-51 Application:poller_HP-WEB-APP:monitor:State returned by Monitor Program (/opt/VRTSvcs/bin/CUSTOM/HP-APP/IXP_RFBH/monitor dbrfbh check):UNKNOWN.
2011/05/25 12:27:21 VCS WARNING V-16-10021-51 Application:ITT0_RICOL-HP-APP-APP:monitor:State returned by Monitor Program (/opt/VRTSvcs/bin/CUSTOM/HP-APP/ITT0_RICOL/monitor):UNKNOWN.
2011/05/25 12:27:22 VCS WARNING V-16-10021-13196 Thread(7) script (/usr/bin/su) terminated due to signal (9) <<<<<<<<<<
2011/05/25 12:27:22 VCS WARNING V-16-10021-62 Application:ITT0_RICOZC-HP-APP-APP:monitor:Abnormal termination of program (/opt/VRTSvcs/bin/CUSTOM/HP-APP/ITT0_RICOZC/monitor).
---------------------------------------------------------------------------
 
3.  Chasing up the status of resources HP-JAVA-APPin engine_A.log
---------------------------------------------------------------------------
2011/05/25 12:27:20 VCS INFO V-16-2-13001 (SYMC-HPUX1) Resource(PDBVAP54-HTC_ISP): Output of the completed operation (monitor)                                            
/usr/lib/hpux32/uld.so: Unable to open '/usr/lib/hpux32/dld.so'.
2011/05/25 12:27:20 VCS ERROR V-16-2-13067 (SYMC-HPUX1) Agent is calling clean for resource(PDBVAP54-HTC_ISP) because the resource became OFFLINE unexpectedly, on its own.
2011/05/25 12:27:20 VCS INFO V-16-1-10299 Resource HP-JAVA-APP(Owner: unknown, Group: HP-APP) is online on SYMC-HPUX1(Not initiated by VCS)                     <<<<
2011/05/25 12:27:20 VCS ERROR V-16-1-10214 Concurrency Violation:CurrentCount increased above 1 for failover group HP-APP
2011/05/25 12:27:20 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group HP-APP on all nodes
2011/05/25 12:27:21 VCS WARNING V-16-10021-51 (SYMC-HPUX1) Application:poller_HP-WEB-APP:monitor:State returned by Monitor Program (/opt/VRTSvcs/bin/CUSTOM/HP-APP/IXP_RFBH/monitor dbrfbh check):UNKNOWN.
2011/05/25 12:27:21 VCS INFO V-16-1-10299 Resource ITT0_RICOI-HP-APP-APP (Owner: unknown, Group: HP-APP) is online on SYMC-HPUX1(Not initiated by VCS)                
---------------------------------------------------------------------------
 
[ Comment ] Resource HP-JAVA-APP is running on the node, SYMC-HPUX2

Environment



 

[ VERSION OF OS/PACKAGE ]
HPUX11v2
SFHA4.1MP2

 


Cause



Etrack 798029 : The offline monitor would fail for Application resources, because the 'su -'  command would fail because the application user's home directory didn't exist to allow an offline monitor to be run.


Solution



 

[ FINDINGS AND SUGGEESTION ]
1. There is the fix for Solaris as per the Internal Technote: 285347 / Point Patch was posted to 4.1MP1_e798029.tar.Z

2. However, backline confirmed that there are no fixes for HPUX11v2.
The symptoms of Etrack 798029 were seen on 4.1mp2, because the su couldn't succeed in the Application resource monitor entry point.
These symptoms can be seen where the application user's home directories are managed on HA mount points also managed by VCS.
 
The offline monitor would fail for Application resources, because the 'su -'  command would fail because the application user's home directory didn't exist to allow an offline monitor to be run.
In this case, the "su -" failed because the maxfiles kernel limit was exhausted for the root user.
This resulted in errors in the messages file 'file table full' and su commands (and other root processes) being killed (dumping core) by the kernel on the problematic system.
 
The concurrency violation appears to have been triggered because of the unexpected failure of the su command. This su operation is removed in the 5.0  release, and a fix for the missing homedir was not ported to the HPUX platform on 4.1mp2, meaning we have no MP/RP/HF fixes available to resolve this on 4.1mp2.
 
As 4.1mp2 is in Partial Support, and END OF SUPPORT in 10 weeks, SYMC are very unlikely to get a fix from VCS engineering, escpecially as this is fixed in our more recently released 5.0MP1 product for HPUX 11.23 IA64.
 

3. So therefore, the best way to fix the problem permanently is to upgrade all nodes to SF/VCS 5.0mp1 at the soonest convenience.



Article URL http://www.symantec.com/docs/TECH161118


Terms of use for this information are found in Legal Notices