How to troubleshoot Scan Engine RPC issues

Article:TECH94315  |  Created: 2009-01-02  |  Updated: 2011-11-16  |  Article URL http://www.symantec.com/docs/TECH94315
Article Type
Technical Solution


Environment

Subject

Issue



You need to know step-by-step instructions on how to troubleshoot issues when Scan Engine is used with a NetApp filer via RPC protocol.


Solution



Scan Engine recommendations

  1. Ensure all steps as per TECH89560 have been applied to your environment.
  2. If possible, set local logging level on SSE scanners to Verbose (if low space available on your Scan Engine scanners, set it to Info).
    Please note you will need to monitor SSE hosts for disk space after this is enabled.


Please provide the following files to Symantec Technical Support:

  1. Note the date and time when the issue was observed, along with additional information which could help Symantec to better identify/locate the issue, such as filenames, paths etc
  2. Filer logs/syslogs from all filers
    - "/etc/messages" from filer
    - "/etc/logs/" folder from filer
  3. Provide the outputs of the following commands which should be run from the Filer's Command Line:
    - "vscan options"
    - "vscan scanners"
    - "netstat -a"
    - "sysstat -c1"
    - "cifs top -s suspicious"
  4. Scan Engine logs from each scanner, make sure to include the .dat log file as well as the log Symantec Scan Engine log file.
  5. Scan Engine scanners' Windows Application and System event logs
  6. All Scan Engine ".xml" files from each scanner

 



If the Symantec Scan Engine service is either crashing or stopping

Configure DrWatson to generate full memory dumps:

 

Windows 2003 Server

  1. Click "start"
  2. Click "run"
  3. Enter "Drwtsn32"
  4. Make sure the "crash dump type" is set to "Full"
  5. The following check boxes should also be enabled:
    - "Dump Symbol Table"
    - "Dump All Thread Contexts"
    - "Append to Existing Log File"
    - "Create Crash Dump File"
  6. Note down the locations where the dump(s) will be stored
  7. Click OK


Windows 2000 Server

  1. Click "start"
  2. Click "run"
  3. Enter "Drwtsn32"
  4. Make sure the following check boxes are enabled:
    - "Dump Symbol Table"
    - "Dump All Thread Contexts"
    - "Append to Existing Log File"
    - "Create Crash Dump File"
  5. Note down the locations where the dump(s) will be stored
  6. Click OK



Install Userdump to collect memory dumps upon service termination

Alternatively, only if instructed by Symantec Technical Support, you can install and configure Microsoft's Userdump utility.

  1. Provide your OS details and software installed along with Scan Engine to the Support Engineer, in order to have a sanity check of the Userdump tool on your system.
  2. Download "User Mode Process Dumper Version 8.1" from: http://www.microsoft.com/downloads/details.aspx?FamilyID=e089ca41-6a87-40c8-bf69-28ac08570b7e&DisplayLang=en
  3. Run the Setup.exe program for your processor.
  4. By default, this Setup.exe program is included with the Userdump.exe tool in the C:\kktools\userdump8.0 folder.
  5. This Setup.exe program installs a kernel-mode driver, installs the Userdump.sys file, and creates the Process Dump icon in Control Panel.
  6. Unless you have a specific need, disable the "dump on process termination" feature when you run the Setup.exe program.
  7. In Control Panel, double-click "Process Dumper".
  8. On the "Process Monitoring" tab, click New, add the appropriate program name (symcscan.exe) to the Monitor list, and then click OK.
  9. When the program stops responding, a memory dump will be generated automatically.
  10. Collect the process dump for analysis.


For more information on User Mode Process Dumper, please refer to the following Microsoft KB article: http://support.microsoft.com/kb/241215

Important:

  1. Remember to disable or uninstall the Process Dumper after obtaining the relevant information for Support.
  2. Do not collect a memory dump unless a Debug build of Symantec Scan Engine is being used and instructions are provided by Symantec Enterprise Technical Support.
  3. Do not check the option "Dump on process termination" on userdump.exe unless instructed by Symantec Enterprise Technical Support.

 



Filer recommendations

If possible, enable Syslog logging on the Filer with all events (error to debug) and all facilities events sent to a Syslog server.



Performance analysis and troubleshooting on Scan Engine and Windows server

Performance Logs
The following steps will guide you to collect performance-related data from the Scan Engine server.

Notes:
- Collect performance logs only from the Symantec Scan Engine scanners mostly affected by the issue you need to troubleshoot/collect logs for.
- Generally, performance logs collection doesn't affect the system's performance. However, you should expect an increase in CPU and disk usage. It is very important that you run the performance log collection only while reproducing the issue you need to troubleshoot.
- Make sure your C: drive has enough free space (at least 2-3 GB) before applying the following steps.


Windows 2003 Server

  1. Enter Windows Computer Management, browse to "System Tools / Performance Logs and Alerts / Counter Logs", then click "Action" menu and "New Log Settings..." as illustrated below:

  2. Enter a name for the new perfmon log:
  3. Click "Add Objects...":
  4. Select "Use local computer counter objects" then select "TCPv4" from the objects list as in the illustration below:

    Click Add then Close.
  5. Once TCPv4 has been added, click "Add Counters...":
  6. First, select the Object called Redirector, then select (keep the CTRL key pressed) the following:

    Click Add
  7. Stay on the Add Counters window, then select the following:

    Click Add then Close
  8. Set sampling to be every 10 seconds:
  9. Move to the "Log Files" tab, then make sure the log file is created as a CSV file:
  10. Move to the "Schedule" tab, then select the following:
  11. If the following dialog is shown, answer YES:
  12. When ready to reproduce the issue you're observing, start the counter by selecting it and clicking the "play" icon.
    The Scan Engine icon will turn from red to green.
  13. After the issue has been reproduced, make sure to stop the collection by clicking the "stop" button:

 


Windows 2000 Server

  1. Enter Windows Computer Management, browse to "System Tools / Performance Logs and Alerts / Counter Logs", then click "Action" menu and "New Log Settings..." as illustrated below:
  2. Enter a name for the new perfmon log:
  3. Click "Add...":
  4. Select "Use local computer counter objects" then select "Network Interface" and counters as in the illustration below:

    Then click Add and Close.
  5. Click "Add..." again:
  6. Select the following options:

    When selecting counters, keep the CTRL key pressed and select the following item towards the end of the list:

    Click Add then Close
  7. Click "Add..." again:
  8. Select the following options:

    Click Add then Close.
  9. Set sampling to be every 10 seconds:
  10. Move to the "Log Files" tab, then make sure the log file is created as a CSV file:
  11. Move to the "Schedule" tab, then select the following:
  12. If the following dialog is shown, answer YES:
  13. When ready to reproduce the issue you're observing, start the counter by selecting it and clicking the "play" icon.
    The Scan Engine icon will turn from red to green.
  14. After the issue has been reproduced, make sure to stop the collection by clicking the "stop" button:


Hide details for Process MonitorProcess Monitor
Note: Process Monitor data collection can generate very large files in some environments.
Please run this tool only if instructed to do so by Symantec Enterprise Technical Support.

  1. Download Procmon from http://download.sysinternals.com/Files/ProcessMonitor.zip
  2. Extract and run the tool on the all the Scan Engine scanners
  3. Stop the log by clicking on the following icon:
  4. Clear the existing log by clicking on the following icon:
  5. Setup logging filter for the "symcscan.exe" process only, by clicking on "Filter" menu and "Filter...", as shown in the steps below:




  6. When ready to start the capture (either before reproducing the problem or when problem is due to occur), click the following icon to start capture:
  7. At the end of collection, save the log file by selecting the options below:

 



What other logs to gather should the problem occur again

Should the problem occur again, please provide the following files to Symantec Technical Support:

  1. Outputs of the following commands which should be run from the Filer's Command Line:
    - "vscan options"
    - "vscan scanners"
    - "netstat -a"
    - "sysstat -c1"
    - "cifs top -s suspicious"
  2. Output of a "netstat -an" command run on the Scan Engine server. Use the command "netstat -an > connections.txt", and send support the connections.txt file.
    Note: this output should be generated when the issue is observed
  3. If the Scan Engine service is crashing, provide full memory dumps obtained through DrWatson
  4. If the Scan Engine service is not crashing, provide a process dump of the "symcscan.exe" process obtained through User Mode Process Dumper while the problem is occurring.
  5. Performance logs from each Scan Engine scanner
  6. Process Monitor log from each Scan Engine scanner
  7. Windows Application and System event logs from each Scan Engine scanner.
  8. If possible, provide a Symbatchdiag file from each Scan Engine scanners (for more information: http://service1.symantec.com/support/ent-security.nsf/docid/2008053007034848)


 



Legacy ID



2009060212572454


Article URL http://www.symantec.com/docs/TECH94315


Terms of use for this information are found in Legal Notices