How to troubleshoot NDMP Backups failures when status code 99 (or other NDMP backup failure) is reported. Includes logging instructions.

Article:TECH56492  |  Created: 2007-01-31  |  Updated: 2014-09-03  |  Article URL http://www.symantec.com/docs/TECH56492
Article Type
Technical Solution


Environment

Issue



How to troubleshoot NetBackup for NDMP Backup failures when status code: 99 (NDMP backup failure) is reported.  Includes logging instructions.
This article can be used for most NDMP status codes.


Error



STATUS CODE: 99


Cause



Status 99 in NDMP backups is generic. Further troubleshooting is required.


Solution



During regular (standard) NDMP backups, avoid potential NDMP communications failures between NetBackup media servers and the Network Attached Storage (NAS) host.
The following troubleshooting steps may help isolate the root cause of NDMP backup issues. If they do not, proceed with logging instructions in Step 9.

NOTE:  Terminology for the types of NDMP backups can be confusing:

  • Remote NDMP:  The backup is performed on the media server, using the device paths defined to the media server.
  • Local:  The backup is performed using the device paths attached/zoned to the NAS host.
  • 3-Way:  The backup of a NAS host is performed using the device paths attached/zoned to another NAS host

To isolate the root cause of a NDMP backup failure:

1. On the media server, test the connection to the NAS device on the NDMP port (10000) by typing the telnet command.  Try both the hostname and its IP address.  For example: telnet ndmp_host 10000

2. To test connectivity and credentials of the NDMP host and its devices:

  •  Type tpautoconf -verify [NAS hostname]
  •  Type tpautoconf -probe [NAS hostname] to  probe the scsi bus for devices (local or 3-Way only).
  •  On another master or media server, type the above two commands to check for another route or network path.
  • Compare the probe results with the native NAS commands:

          NetApp: sysconfig -t
          EMC Celerra: server_devconfig [datamover name] -l -s -n  (list, scsi, non-disk)
         Other:  See system documentation

3. On the NAS, and type the command ndmpd status to verify that the NDMP daemon is running.  If it is not, enable the ndmp daemon and verify again by typing the command ndmpd status.  For information on starting the ndmp daemon, see the NAS system documentation.

4. For EMC Celerra devices, ensure that the SnapSure feature is enabled.  See TECH170310 for information on SnapSure and other considerations.

5. On the media or master server, type the telnet and tpautoconf command tests again from Step 1 to test connectivity, credentials, and devices.

6.  If Local or 3-Way backup, do the following to verify that the volume can be accessed and backed up:

  • First, mount a scratch tape in the drive using robtest or some other means and perform a backup.
  • On the NAS device, type the appropriate native command:

Network Appliance (NetApp): dump 0uf nrst0a /<file system> (Substitute the proper non-rewind device path if it is not nrst0a as in this example).
Note:  For Cluster-mode NetApp, refer to the NetApp Document 'Clustered Data ONTAP 8.2 Command Map for 7-Mode Administrators' as the dump command does not work for C-Mode volumes.
Excerpt:

Understanding the different clustered Data ONTAP shells for CLI commands
A cluster has three different shells for CLI commands:

• The clustershell is the native shell, started automatically when you log in to the cluster.
It provides all the commands you need to configure and manage the cluster.
• The nodeshell is a special shell that lets you run a subset of 7-Mode commands.
These commands take effect only at the node level. You can switch from the clustershell to a nodeshell session to run nodeshell commands interactively, or you can run a single nodeshell command from the clustershell. You will know a command is a nodeshell command if it has the (long) form:

system node run -node {nodename|local} commandname

This command will write data from the specified file system on the Network Appliance server directly to tape. If this dump does not complete successfully, then the problem is most likely a NetWork Appliance problem. To resolve the issue, open a call with Network Appliance technical support. If the dump does complete successfully, open a call with NetBackup support.

For EMC Celerra, refer to the server_archive command in the system documentation.

NOTE:
  The Backup Path specified in the NetBackup policy must be the exact case-sensitive path as the filer sees it.

7.  For performance issues, try backing up the path in the policy to a null device and note the timing:

NetApp:  dump 0f null /vol/volname
EMC Celerra:  server_archive <datamover name> -w -f /dev/null/ -J <file path>

8. Based on your findings, proceed with any required configuration changes or, if there are no changes, begin backups or restores.

9.  If none of the above troublesooting steps resolve the issue, enable debug logging on the NetBackup media server and on the NAS device.

To enable debug logging:

1. Depending on your media server platform, do the following:

For Windows media server: 

  • On the NetBackup console, go to Host Properties --> Media Server and double-click the media server that services this backup to open the Properties dialog box. If this media server is the same as the master server, go to Master Server instead.
  • Go to the Logging section and change the global logging level to 5 (maximum).
  • Click OK to exit.

For UNIX media server: 

  • Add VERBOSE to the /usr/openv/netbackup/bp.conf file in the following format:  VERBOSE = 5

2. Make sure that the following legacy log directories exist on the media server that services the backup:

For Windows media server:
\NetBackup\logs\bpbrm
\NetBackup\logs\bptm
\NetBackup\logs\ndmpagent

For Unix media server:
/usr/openv/netbackup/logs/bpbrm
/usr/openv/netbackup/logs/bptm
/usr/openv/netbackup/logs/ndmpagent

3. Run the following commands on the media server that services this backup to increase the unified logging levels:

For Windows media server:

<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6 -s DiagnosticLevel=6
<install path>\NetBackup\bin\vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6 -s DiagnosticLevel=6

For Unix media server:

/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 134 -s DebugLevel=6 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -a -p 51216 -o 151 -s DebugLevel=6 -s DiagnosticLevel=6

4. Enable ndmpd debug logging on the NAS device:

  • For Network Appliance NAS, type the following to enable ndmpd debug logging:  ndmpd debug 70
    NOTE:  For OnTap 8.1 and above, the commands are different. Refer to TECH178502 for 7-Mode or TECH218460 for Cluster Mode (C-Mode).
    OS version of any filer can be determined by viewing the output of tpautoconf -verify [NAS host name]
    If necessary, further ndmpd event logging can be enabled.  Refer to the ndmpd.backup.log.enable command in this Network Appliance Support article.
  • EMC Celerra:  Enable debug logging on the datamover as described in Article TECH150646
  • For other NAS devices, see the vendor documentation.

5. Reproduce the error.

6.  Disable logging:

  • For Network Appliance NAS, type the following to disable ndmpd debug logging:  ndmpd debug 0
    Refer to the NetApp articles listed in Step 4 for OnTap 8.1 and above.
  • EMC Celerra, see the disable commands in TECH150646.
  • For other NAS devices, see the vendor documentation.

7. Retrieve the bpbrm, bptm, ndmpagent logs from the NetBackup media server for today's date. 

8. Do one of the following to retrieve logs from the NAS device:

  • Pull the datamover server_log from the EMC Celerra:  server_log server_2 -a -s > /output.file  (assumes ‘server_2’ is the datamover name)
  • Pull the system log, backup log, and the ndmpdlog from the Network Appliance NAS:
    • /vol/etc/messages
    • /vol/etc/logs/backup
    • <volume>/etc/log/ndmpdlog (/vol/etc/log/mlog in DataOnTap 8.x)
    • For remote access to the files, try windows browser  (\\Filer_Name\C$\etc\File_Name)

9. Type the commands on the media server to disable logging:

For Windows media server:

<install path>\NetBackup\bin\vxlogcfg -r -p 51216 -o 134 -s DebugLevel=1 -s DiagnosticLevel=6
<install path>\NetBackup\bin\vxlogcfg -r -p 51216 -o 151 -s DebugLevel=1 -s DiagnosticLevel=6 

For Unix media server:

/usr/openv/netbackup/bin/vxlogcfg -r -p 51216 -o 134 -s DebugLevel=1 -s DiagnosticLevel=6
/usr/openv/netbackup/bin/vxlogcfg -r -p 51216 -o 151 -s DebugLevel=1 -s DiagnosticLevel=6


10. Create a temp directory.
 For example: C:\temp\vxul or /tmp/vxul

11. Run the following command to retrieve all unified logs for today's date:

For Windows media server: <install path>\NetBackup\bin\vxlogmgr -c -t 1 -f c:\temp\vxul

For Unix meda server: /usr/openv/netbackup/bin/vxlogmgr -c -t 1 -f /tmp/vxul

Where the '-t 1' value will retrieve the unified logs from the last hour.  Adjust the value as appropriate to go back past the beginning of the job.

12. Compress the temp directory (Zip for Windows, tar for Unix) and send to Symantec NetBackup Support along with the legacy logs and the NAS logs.

13. To create a long listing of the policy, run the following comand on the NetBackup master server: 

Windows master server: <install path>\NetBackup\bin\admincmd\bppllist <name of policy> -L

Unix master server: /usr/openv/netbackup/bin/admincmd/bppllist <name of policy> -L

14. On the media server, on the command line, run the nbsu tool as follows to create the output file:

Windows media server: 

  • In a command window, change the directory to <install path>\NetBackup\bin\support
  •  Type the following command:  nbsu -c -t -nozip

Unix media server: 

  • Type the following command: /usr/openv/netbackup/bin/support/nbsu -c -t

The script will create an output file and tell you where it was stored.

15. Cut and paste the Job Details for the job in the Activity Monitor during the backup attempt.  Send that to Symantec Support along with the nbsu output file and the bppllist output. 

Error when adding credentials for NDMP host:

Unable to validate the filer wide credentials. NDMP failed to verify host (58).

Ensure the ndmp userid is created with MD5 encryption.  For Network Appliance, it can be switched between plaintext and MD5.  Refer to this article.
For EMC, use MD5.  Search for the document 'Adding an NDMP user in the NetBackup Admin Console' on the EMC Support website (requires an account).

 


Supplemental Materials

Value99
Description

NDMP backup failure



Legacy ID



295366


Article URL http://www.symantec.com/docs/TECH56492


Terms of use for this information are found in Legal Notices