Application agent falsely detect wac process as offline even when the process is running properly. Application agent is killing the wac application and restarting it.

Article:TECH77390  |  Created: 2009-01-29  |  Updated: 2012-10-15  |  Article URL http://www.symantec.com/docs/TECH77390
Article Type
Technical Solution


Environment

Issue



Application agent falsely detect wac process as offline even when the process is running properly. Application agent is killing the wac application and restarting it.


Solution



Overview:
 
Application agent falsely detect wac process as offline even when the process is running properly.  Application agent is killing the wac application and restarting it.

 
Details:
 
Application agent is detecting wac process as offline and kills it by calling clean entry point when wac process is still active
 

 
Exact Error Message from debug log:
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:Total number of Processes specified:1.
 
Application.C:Application_monitor[227]
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:Checking for Process:/opt/VRTSvcs/bin/wac as Default User:root.
 
Application.C:Application_monitor[246]
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:Process:/opt/VRTSvcs/bin/wac; User:root.
 
Application.C:processExists[452]
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:New process string after removing extra space:/opt/VRTSvcs/bin/wac; User:root.
 
Application.C:processExists[491]
 
2009/07/14 12:40:19 VCS DBG_4 V-16-50-0 Application:wac:monitor:Command prepared for getting pid is
 
/bin/ps --cols=100000 --User=root -ao pid,args | /bin/grep "/opt/VRTSvcs/bin/wac" | /bin/grep -v /bin/grep | /usr/bin/tr -s " " " " | /bin/sed -e 's/^ //' | /bin/cut -f1 -d" ".
 
Application.C:processExists[519]
 
2009/07/14 12:40:20 VCS DBG_4 V-16-50-0 Application:wac:monitor:pidstring:7473
 
2009-07-14 12:40:20, process:/opt/VRTSvcs/bin/wac.
 
Application.C:getPidOfArg[908]
 
2009/07/14 12:40:20 VCS DBG_4 V-16-50-0 Application:wac:monitor:Calling getArgsOfPid for pid:7473 for process:/opt/VRTSvcs/bin/wac.
 
Application.C:getPidOfArg[932]
 
2009/07/14 12:40:21 VCS DBG_4 V-16-50-0 Application:wac:monitor:Calling VCSAgExec for pid:7473
 
Application.C:getArgsOfPid[874]
 
2009/07/14 12:40:21 VCS DBG_4 V-16-50-0 Application:wac:monitor:for pid:7473, arguments are:/opt/VRTSvcs/bin/wac
 
Application.C:getArgsOfPid[879]
 
2009/07/14 12:40:21 VCS DBG_4 V-16-50-0 Application:wac:monitor:Calling getArgsOfPid for pid:2009-07-14 12:40:20 for process:/opt/VRTSvcs/bin/wac.
 
Application.C:getPidOfArg[932]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:Calling VCSAgExec for pid:2009
 
Application.C:getArgsOfPid[874]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:for pid:2009, arguments are:
 
Application.C:getArgsOfPid[879]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:pid not found for process:/opt/VRTSvcs/bin/wac.
 
Application.C:getPidOfArg[950]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:process(/opt/VRTSvcs/bin/wac) does not exist.
 
Application.C:processExists[609]
 
2009/07/14 12:40:22 VCS DBG_4 V-16-50-0 Application:wac:monitor:Process:/opt/VRTSvcs/bin/wac; return state: Offline.
 
Application.C:Application_monitor[257]
 
2009/07/14 12:40:22 VCS ERROR V-16-2-13067 Thread(4135934880) Agent is calling clean for resource(wac) because the resource became OFFLINE unexpectedly, on its own.
 
2009/07/14 12:40:25 VCS ERROR V-16-2-13068 Thread(4135934880) Resource(wac) - clean completed successfully.
 
2009/07/14 12:40:25 VCS ERROR V-16-2-13073 Thread(4135934880) Resource(wac) became OFFLINE unexpectedly on its own. Agent is restarting (attempt number 1 of 3) the resource.
 

 
getArgsOfPID = pid:2009-07-14 12:40:20 for process:/opt/VRTSvcs/bin/wac.
 

 
Log Files:
 
/var/VRTSvcs/lock/tmp/Application-1 file
 
Troubleshooting:
 
While calling any entry point agent framework redirects the output file descriptor to a file (thread specific file), so that output of the entry point can be captured. In case of Application agent, online entry point invokes the program mentioned in StartProgram attribute. As output file descriptor is redirected to thread specific file when invoking the online script and online script invokes the StartProgram, so output file descriptor of StartProgram will also point to the same file.
 
Problem occurs when the same thread which executed the online entry point executes monitor entry point.
 
In monitor entry point of the Application agent, we query the process id of the running process from the process table. The output will be logged into the thread specific file and if in mean time the background process write some thing to the output file descriptor then that will get written to the same file (thread specific file).
 
Resolution
 
Fix the StartProgram, StopProgram, MonitorProgram attribute of all the Application resources in one of the following two ways:
 
  (a) Redirect the stdout and stderr of StartProgram, StopProgram, MonitorProgram process to /dev/null, or
 
  (b) Redirect the stdout and stderr of StartProgram, StopProgram, MonitorProgram process to some file
 

 

Supplemental Materials

SourceETrack
Value1837979
Description

getArgsOfPid for pid:2009-07-14 12:40:20 for process:/opt/VRTSvcs/bin/wac. Application.C



Legacy ID



337904


Article URL http://www.symantec.com/docs/TECH77390


Terms of use for this information are found in Legal Notices