3RD PARTY: NetBackup Services are randomly shutting down on Windows servers after applying a patch for McAfee McShield 8.5 or 8.7i.

Article:TECH56658  |  Created: 2009-01-04  |  Updated: 2013-08-22  |  Article URL http://www.symantec.com/docs/TECH56658
NOTE: If you are experiencing this particular known issue, we recommend that you Subscribe to receive email notification each time this article is updated. Subscribers will be the first to learn about any releases, status changes, workarounds or decisions made.
Article Type
Technical Solution

Product(s)

Environment

Issue



3RD PARTY: NetBackup Services are randomly shutting down on Windows servers after applying a patch for McAfee McShield 8.5 or 8.7i.


Solution



Vendor/Product:
McAfee McShield 8.5 patch 3 and newer as well as 8.7i

Detail/Symptom(s):
 
NetBackup services randomly shut down including:
 
  • NetBackup Resource Broker Service (nbrb.exe)
  • NetBackup Notification Service (nbnos.exe)
  • NetBackup Policy Execution Manager Service (nbpem.exe)
  • NetBackup Service Layer Service (nbsl.exe)
Active Jobs finish but tapes are not moved from drives back to slots
 
Active Jobs which need to span media sit at Waiting for next media: Any
 
Queued Jobs do not go active
 

 
The above symptoms can happen once or twice per day and can occur on the smallest or largest installations.  The NetBackup services are shutting themselves down because inter-process sockets are being disconnected.  The processes attempt to reconnect, but are unable to do so, and as a result the processes and services shut down.  

These symptoms have been seen on systems running McAfee McShield 8.5 patches 3 and newer and 8.7i with NetBackup 6.0 (all patches) and NetBackup 6.5 (all patches).

Log Files:
12/20/07 01:53:56.227 137 PID:7184 TID:7556 [TAO] ACE_Select_Reactor_Notify::notify [handle=0x1f8]: write to notification pipe handle failed: An existing connection was forcibly closed by the remote host. (10054)
12/20/07 01:53:56.227 137 PID:7184 TID:7556 [TAO] sleep_hook failed: An existing connection was forcibly closed by the remote host.
12/20/07 01:53:56.242 137 PID:7184 TID:920 [TAO] handle_notify_pipe_close - taking action REOPEN
12/20/07 01:54:17.336 137 PID:7184 TID:920 [TAO] handle_notify_pipe_close: failed to re-open notification pipe: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.


Workarounds:
There are several workarounds, any of which can be implemented:
1. Uninstall McAfee McShield 8.5 (or 8.7i) and reboot.  (Simply stopping McShield services is not sufficient.)
2. Roll-back to McAfee McShield 8.0.
3. Rename the McAfee Anti-Virus Mini-Firewall Driver file C:\WINDOWS\system32\drivers\MFETDIK.sys and reboot.  Note that this also removes the functionality provided by this file (port blocking access protection rules and identification of source IP Address for a remote attacker).

Best Practice:
In addition to the workarounds listed above, it is always a best practice to configure McAfee by accounting for NetBackup files in three areas of McAfee.

1. Add NetBackup processes to McAfee's Low-Risk Processes list.  (Master Servers, Media Servers)
2. Add NetBackup directories to McAfee's Exclude list. (Master Servers, Media Servers)
3. Uncheck the McAfee setting Scan files opened for Backup. (Master Servers, Media Servers, Clients)

How to add critical NetBackup Master Server and Media Server processes to McAfee's Low-Risk Processes List.
This same process can be used to add exclusively the bpinetd.exe and bpbkar32.exe process on machines that only run the NetBackup Client Service.

1. Launch the McAfee VirusScan Console.
2. Right-click on On-Access Scanner and select Properties:
 

3. Navigate to All Processes > Processes tab.
4. Switch the radio button to "Use different settings for high-risk and low-risk processes:"
 

5. Navigate to Low-Risk Processes > Processes tab > click Add > click Browse:
 

6. Process by process - add this list of NetBackup processes to the list of Low-Risk Processes:

Processes located in <install_path>\VERITAS\Volmgr\bin\
avrd.exe - Automatic Volume Recognition Daemon
ltid.exe - NetBackup Device Manager Service
vmd.exe - NetBackup Volume Manager Service

Processes located in <install_path>\VERITAS\NetBackup\bin\
bpbkar32.exe - NetBackup Backup Engine
bpbrm.exe - NetBackup Backup and Restore Manager
bpcd.exe - NetBackup Connection Daemon
bpcompatd.exe - NetBackup Compatibility Service
bpdbm.exe - NetBackup Database Manager Service
bpdm.exe - NetBackup Disk Manager
bpinetd.exe - NetBackup Client Service
bpjava-msvc.exe - NetBackup Java Authentication Service
bpjobd.exe - NetBackup Job Daemon
bprd.exe - NetBackup Request Manager Service
bptm.exe - NetBackup Tape Manager
nbconsole.exe - NetBackup Administration Console
nbemm.exe - NetBackup Enterprise Media Manager Service
nbevtmgr.exe - NetBackup Event Manager
nbjm.exe - NetBackup Job Manager Service
nbnos.exe - NetBackup Notification Service
nbpem.exe - NetBackup Policy Execution Manager Service
nbproxy.exe - NetBackup Proxy process
nbrb.exe - NetBackup Resource Broker Service
nbrmms.exe - NetBackup Remote Manager and Monitor Service
nbsl.exe - NetBackup Service Layer Service
nbstserv.exe - NetBackup Storage Lifecycle Manager Service
nbsvcmon.exe - NetBackup Service Monitor Service
nbvault.exe - NetBackup Vault Manager Service
tar32.exe - NetBackup Restore Engine

Additional Processes:
<install_path>\VERITAS\NetBackupDB\WIN32\dbsrv9.exe - Adaptive Server Anywhere - VERITAS_NB Service

C:\Program Files\VERITAS\VxPBX\bin\pbx_exchange.exe - Veritas (Symantec) Private Branch Exchange Service

C:\Program Files\VERITAS\VxPBX\bin\monitor_server.exe - Veritas process used when NBU is highly available

7. Once all of the above processes have been added, with Low-Risk Processes selected, select the Detection tab and uncheck When writing to disk and When reading from disk:
 


How to add NetBackup paths to McAfee's list of what not to scan:

1. Within On-Access Scan Properties, select Default Processes on the left column, then select the Detection tab.  Click on Exclusions for the category of What not to scan.

 

2. Click Add and individually browse out to these three locations adding each in turn:

...\Veritas\Volmgr\*                             (be sure to append * to the path once each path has been added)
...\Veritas\NetBackup\*                       (be sure to append * to the path once each path has been added)
C:\Program Files\VERITAS\VxPBX\* or C:\Program Files (x86)\VERITAS\VxPBX*   (be sure to append * to the path once each path has been added)

 

3. Within each path excluded, be sure Also exclude subfolders, On read and On write are all checked.
 


How to configure McAfee to not scan files open for backup:

1. Under Default Processes, Low-Risk Processes and High-Risk Processes, click on the Advanced Tab and uncheck Scan files open for backup:
 

Note:  Any machine - master server, media server or client - which is running McAfee should have its McAfee properties modified to disable Scan files opened for Backup.  NetBackup honors the API which is tied to this setting.  Each machine with a NetBackup client installed should be individually modified with this setting unless centralized changes can be made to all clients from a single location (for example, by using McAfee's Event Policy Orchestrator).

For additional information on these settings, please reference McAfee source material:

 http://mysupport.mcafee.com/Eservice/templatepage.aspx?sURL=3

KB55139
Understanding High-Risk, Low-Risk, and Default processes configuration and usage

KB58692
Creating Low-Risk Process exclusions in VirusScan Enterprise

KB55898
Understanding VirusScan Enterprise Exclusions


RESOLUTION:
McAfee has a resolution for the interference introduced by the Mini-Firewall driver mfetdik.sys

 
McShiled 8.5 - HotFix HF482720 (11 August, 2009)   This HotFix is not publicly available.  A call must be placed to McAfee Support to request the HotFix.
 

 
McShield 8.7i - The fix is included in Patch 2 through 4 which are released.
The download page for software updates can be found at McAfee Updates
  The HotFix and Patch 4 address several issues, one of which is this:
 

 
Issue:
 
In high I/O environments where Access Protection is enabled, a performance degradation symptom could be encountered, appearing as a hang. Internal processing by VirusScan drivers was occurring in a serialized fashion, contributing to a bottleneck when large volumes of I/O are being filtered.
 

 
Resolution:
 
The link and mini-firewall drivers will no longer cause a sequential release of objects containing gathered information on the I/O request.
 

 
Note:. Please follow all of McAfee's patch application instructions and reboot recommendations when applying patches.

 

 

 

 

 




Legacy ID



295599


Article URL http://www.symantec.com/docs/TECH56658


Terms of use for this information are found in Legal Notices