PROBLEM DESCRIPTION:
cssd resource not probed in a Oracle RAC 10.2.0.3 environment

SYMPTOMS:

-- RESOURCES NOT PROBED
-- Group Type Resource System

D cvm Application cssd <hostname>

The following command when executed manually appears to hang but will finish after an extended wait period.

# $CRS_HOME/bin/crsctl check cssd
Failure 1 contacting CSS daemon


Removing the following piece of code from the cssd-monitor script allowed the cssd resource to be probed and the cvm group to be brought online:


else
# sometimes, pid file has wrong pid. so if this check passes,
# cssd if assumed healthy
$CRS_HOME/bin/crsctl check cssd 1>$NULL 2>$NULL
if [ $? -eq 0 ]; then
NOCSSD=0
fi



CAUSE:
Oracle BUG 6356744:

The crsctl command is hanging because it is parsing each css*.log file in the /opt/support/home/oracle/crs/log/'hostname'/client directory. There are 110,000+ of these logs. They go back as far as 2007. How are these logs created? Each time the 'crsctl check cssd' command is executed it drops a file in that directory. As mentioned in Oracle BUG 6356744, too many of these css*.log files can cause severe performance issues.



SOLUTION:
The recommendation is execute the following command now and perhaps via a monthly cronjob to clear out these files.

# find $CRS_HOME/log/`hostname`/client -mtime +30 -exec rm -f {} \;

Once these file's have been cleaned up, the 'crsctl check cssd' command should not hang thus allowing the cssd-monitor to complete successfully.