NetBackup status code: 156

Article:HOWTO35032  |  Created: 2010-10-29  |  Updated: 2010-10-29  |  Article URL http://www.symantec.com/docs/HOWTO35032
Article Type
How To


Environment

Subject


NetBackup status code: 156

Message: snapshot error encountered

Explanation: The following are possible causes of this status code:

  • An Enterprise Vault Agent-related error. The following errors can result in a status code 156:

    • VSS_E_BAD_STATE snapshot error

    • VSS_E_INSUFFICIENT_STORAGE snapshot error

  • A snapshot-backup related error regarding Windows Open File Backup or Snapshot Client.

Recommended Action: Do the following, as appropriate:

  • For the Enterprise Vault Agent:

    See the Troubleshooting section of the NetBackup for Enterprise Vault Agent Administrator's Guide.

  • For a Windows Open File Backup Snapshot Provider that uses VSS:

    See the Troubleshooting section of one of the following guides:

    • NetBackup for VMware Administrator's Guide

    • NetBackup for Hyper-V Administrator's Guide

    • NetBackup Snapshot Client Administrator's Guide

    The VSS cache files may be too small for the number of files being backed up using VSS.

    If bpbkar debug logs are turned on, a message similar to the following appears in the bpbkar debug log for the backup.

    8:51:14.569 AM: [1924.2304] <2> tar_base::V_vTarMsgW: ERR - 
    failure reading file: D:\ test.file (WIN32 5: Access is denied. 
    )
    8:51:14.569 AM: [1924.2304] <4> tar_base::V_vTarMsgW: INF - tar 
    message received from dos_backup::tfs_readdata
    8:51:14.569 AM: [1924.2304] <2> tar_base::V_vTarMsgW: ERR - 
    Snapshot Error while reading test.file
    8:51:14.569 AM: [1924.2304] <4> tar_base::V_vTarMsgW: INF - tar 
    message received from tar_backup::nextfile_state_switch
    8:51:14.569 AM: [1924.2304] <2> tar_base::V_vTarMsgW: FTL - 
    Backup operation aborted!
    8:51:14.569 AM: [1924.2304] <2> tar_base::V_vTarMsgW: INF - 
    Client completed sending data for backup
    8:51:14.569 AM: [1924.2304] <2> tar_base::V_vTarMsgW: INF - EXIT 
    STATUS 156: snapshot error encountered

    To increase the VSS cache size by using the Shadow Copy configuration in Windows 2003, do the following in the order listed:

    • In Windows, right-click My Computer and select Manage.

    • In the console tree, right-click Shared Folders, select All Tasks, and select Configure Shadow Copies.

    • Select the volume where you want to make changes, and then select Settings.

    • In the Settings dialog box, change the Maximum Size setting to either of the following: No Limit or a size large enough to suit the requirements of your installation and your usage of VSS.

  • For backups using Snapshot Client and the NAS_Snapshot method, with or without SnapVault:

    • If the backup fails with status code 156, consult the bpfis legacy log, in /usr/openv/netbackup/logs (UNIX and Linux) or install_path\NetBackup\logs (Windows). If the bpfis directory does not already exist, you must create it and rerun the job.

      If necessary, increase the logging level and retry the job.

    • On Windows clients, when restoring files from a backup that is made with the NAS_Snapshot method, log into the NetBackup Client Service as the Administrator account, not as the local system account. Otherwise, the backup fails with status 156.

      • In Windows Services, double-click the NetBackup Client Service.

      • Then check the Log On tab: if the service is not logged on as Administrator, stop the service.

      • Change the logon to the Administrator account and restart the service.

      • Retry the restore.

  • For other NetBackup Snapshot Client issues:

    • The file system that is specified as a snapshot source is not mounted. In this case, you may see the following in the /usr/openv/netbackup/logs/bpfis log:

      17:12:51 bpfis: FTL - snapshot creation failed, status 156
      17:12:51 bpfis: INF - EXIT STATUS 156: snapshot error 
      encountered

      and the following in the /usr/openv/netbackup/logs/bpfis log:

      17:12:51 onlfi_vfms_logf: INF - cannot snap_on, err: 5
      17:12:51 delete_mount_point: INF - Deleted mount point 
      /tmp/__jody_test:20958
      17:12:51 onlfi_freeze: FTL - VfMS error 11; see following 
      messages:
      17:12:51 onlfi_freeze: FTL - Fatal method error
      17:12:51 onlfi_freeze: FTL - vfm_freeze: method: nbu_snap, type: 
      FIM, function: nbu_snap_freeze
      17:12:51 onlfi_freeze: FTL - VfMS method error 5; see following 
      message:
      17:12:51 onlfi_freeze: FTL - nbu_snap_freeze: Cannot turn on 
      snapshot; snapshot source=/opt, cache=/dev/rdsk/c1t3d1s0, snap 
      error=5
      17:12:51 onlfi_thaw: WRN - / is not frozen

      Make sure that the file system that is specified for the snapshot source has been mounted.

    • The file system that is specified as the snapshot source does not correspond to the file system that contains the actual files (as opposed to symbolic links to the files). The mounted file system for the snapshot source must contain the actual files, not symbolic links. If items in the file list, such as /oracle, is a symbolic link to /export/home/oracle, the snapshot source must specify /export, or /export/home, not /oracle.

    • VxVM is selected as the snapshot method but the snapshot source is not configured over a Veritas Volume Manager (VxVM) volume. In this case, you may see the following in the /usr/openv/netbackup/logs/bpfis log:

      17:12:51 bpfis: FTL - snapshot creation failed, status 156
      17:12:51 bpfis: INF - EXIT STATUS 156: snapshot error 
      encountered

      and something like the following in the /usr/openv/netbackup/logs/bpfis log:

      17:12:51 onlfi_vfms_logf: INF - vxvm_freeze: Snapshot source 
      /cockpit1 on device /dev/dsk/c1t0d0s6 is not on a VxVM volume
      17:12:51 delete_mount_point:   INF - Deleted mount point 
      /tmp/_cockpit1_coc_group1:3518
      17:12:51 onlfi_freeze: FTL - VfMS error 11; see following 
      messages:
      17:12:51 onlfi_freeze: FTL - Fatal method error
      17:12:51 onlfi_freeze: FTL - vfm_freeze: method: vxvm, type: 
      FIM, function: vxvm_freeze
      17:12:51 onlfi_freeze: FTL - VfMS method error 9; see following 
      message:
      17:12:51 onlfi_freeze: FTL - vxvm_freeze: Snapshot source 
      /cockpit1 on device /dev/dsk/c1t0d0s6 is not on a VxVM volume
      17:12:51 onlfi_thaw: INF - fim=vxvm
      17:12:51 onlfi_thaw: WRN - /cockpit1 is not frozen

      Make sure that the snapshot source is configured over a Veritas Volume Manager (VxVM) volume.

    • VxVM was selected as the snapshot method, but a Veritas Volume Manager snapshot mirror of the snapshot source volume had not been created before running the backup, or another backup is currently running that uses the snapshot mirror. In either case, you may see the following in the /usr/openv/netbackup/logs/bpfis log:

      17:12:51 onlfi_freeze: FTL - VfMS error 11; see following 
      messages:
      17:12:51 onlfi_freeze: FTL - Fatal method error
      17:12:51 onlfi_freeze: FTL - vfm_freeze: method: vxvm, type: 
      FIM, function: vxvm_freeze
      17:12:51 onlfi_freeze: FTL - VfMS method error 3; see following 
      message:
      17:12:51 onlfi_freeze: FTL - find_ready_snapshot: Cannot find 
      available snapshot mirror

      Refer to the NetBackup Snapshot Client Administrator's Guide for information on how to create a snapshot mirror on the client before you run the backup.

    • VxVM was selected as the snapshot method, and a Veritas Volume Manager snapshot mirror of the snapshot source volume has been created. However, two different backup jobs (A and B) try to back up the same volume (for example, vol01), but job A starts before job B. After an available snapshot mirror is found, a brief pause occurs before the snapshot is formed. Job B that runs slightly behind job A may try to create a snapshot of the snapshot mirror immediately before job A creates the snapshot and gets the lock on it.

      In this case, you may see the following in the /usr/openv/netbackup/logs/bpfis log:

      17:12:51 onlfi_freeze: FTL - VfMS error 11; see following 
      messages:
      17:12:51 onlfi_freeze: FTL - Fatal method error
      17:12:51 onlfi_freeze: FTL - vfm_freeze: method: vxvm, type: 
      FIM, function: vxvm_freeze
      17:12:51 onlfi_freeze: FTL - VfMS method error 3; see following 
      message:
      17:12:51 onlfi_freeze: FTL - vxvm_freeze: Command failed with 
      status=11: /usr/sbin/vxassist -g rootdg snapshot vol01 
      VfMSCAAu7a4Uw </dev/null>/var/tmp/VfMSAAAs7a4Uw 
      2>/var/tmp/VfMSBAAt7a4Uw

      The job that was unable to get a lock (job B in the preceding example) fails, and must be run again.

    • When using nbu_snap as a snapshot method, you may have stale snapshots if status code 156 occurs with the following messages in the /usr/openv/netbackup/logs/bpfis log. (Stale snapshots are those that nbu_snap did not automatically delete.)

      17:12:51 onlfi_freeze: FTL - VfMS error 11; see following 
      messages:
      17:12:51 onlfi_freeze: FTL - Fatal method error
      17:12:51 onlfi_freeze: FTL - vfm_freeze: method: nbu_snap, type: 
      FIM, function: nbu_snap_freeze
      17:12:51 onlfi_freeze: FTL - VfMS method error 5; see following 
      
      message:
      17:12:51 onlfi_freeze: FTL - nbu_snap_freeze: Cannot turn on 
      snapshot; snapshot source=/oracle/ufs_r, 
      cache=/dev/rdsk/c4t1d11s4,snap error=11
    • Look for stale snapshots by running the /usr/openv/netbackup/bin/driver/snaplist command when there are no active backups running. If the snaplist command shows cache entries, there are stale snapshots. Nothing is displayed if there are no stale snapshots.

      Example snaplist output:

      id  ident        size     cached    minblk     err time
      43  6515    8390970    0          0          0  11/16/00 13:31:36
      device = /dev/rdsk/c1t6d0s0
      cache  = /dev/rdsk/c1t6d0s7
    • Use the snapoff command to remove the stale snapshot, as follows:

      /usr/openv/netbackup/bin/driver/snapoff id

      where id is the ID from the snaplist output (such as 43 in the preceding example).

    • If a backup using the VxFS_Checkpoint snapshot method failed, the NetBackup bpbkar process should automatically remove the clone. Sometimes, however, bpbkar is unable to remove the clone. In this case, you may see messages such as the following in the /usr/openv/netbackup/logs/bpfis log:

      15:21:45.716 [4236] <4> create_mount_point: INF - Created mount 
      point /tmp/_vtrax_test:4236
      15:21:45.869 [4236] <2> onlfi_vfms_logf: INF - vxfs clone handle 
      : 9600344
      15:21:45.870 [4236] <2> onlfi_vfms_logf: INF - 
      VxFS_Checkpoint_freeze: Cannot create checkpoint; status=17
      15:21:45.872 [4236] <4> delete_mount_point: INF - Deleted mount 
      point /tmp/_vtrax_test:4236
      15:21:45.873 [4236] <32> onlfi_freeze: FTL - VfMS error 11; see 
      following messages:
      15:21:45.873 [4236] <32> onlfi_freeze: FTL - Fatal method error 
      was reported
      15:21:45.873 [4236] <32> onlfi_freeze: FTL - vfm_freeze: method: 
      VxFS_Checkpoint, type: FIM, function: VxFS_Checkpoint_freeze
      15:21:45.873 [4236] <32> onlfi_freeze: FTL - VfMS method error 
      17; see following message:
      15:21:45.874 [4236] <32> onlfi_freeze: FTL - 
      VxFS_Checkpoint_freeze: Cannot create checkpoint; status=17

      Remove the clone as follows.

      Note:

      If the checkpoint is not removed, you cannot use VxFS_Checkpoint to back up any data in the file system where the checkpoint is mounted.

    • List the name of the checkpoint by entering the following VxFS command:

      /usr/lib/fs/vxfs/fsckptadm list /file_system

      where file_system is the name of the file system where the checkpoint is mounted. A sample output follows. In this example, /vtrax_test is the file system and fi_ckpt is the name of the checkpoint.

      /vtrax_test
      fi_ckpt:
      ctime = Mon Nov 12 10:08:13 2001
      mtime = Mon Nov 12 10:08:13 2001
      flags = largefiles
    • Remove the checkpoint by entering the following:

      /usr/lib/fs/vxfs/fsckptadm remove checkpoint /file_system
    • If the checkpoint cannot be removed, unmount the checkpoint and retry the first step in this procedure.

    • If a snapshot backup fails using TimeFinder, ShadowImage, or BusinessCopy method, there may be a VxVM clone left over from a previous backup. You may see messages similar to the following in the /usr/openv/netbackup/logs/bpfis log:

      19:13:07.686 [14981] <2> onlfi_vfms_logf: INF - do_cmd: Command 
      failed with status=20: /usr/openv/netbackup/bin/bpdgclone -g 
      wil_test -n vol01 -f /var/tmp/HDSTFCAAs7aOqD </dev/null 
      >/var/tmp/VfMSAAAq7aOqD 2>/var/tmp/VfMSBAAr7aOqD
      19:13:07.687 [14981] <2> onlfi_vfms_logf: INF - --- Dumping file 
      /var/tmp/VfMSAAAq7aOqD (stdout):
      19:13:07.687 [14981] <2> onlfi_vfms_logf: INF - --- End of file 
      /var/tmp/VfMSAAAq7aOqD
      19:13:07.687 [14981] <2> onlfi_vfms_logf: INF - --- Dumping file 
      /var/tmp/VfMSBAAr7aOqD (stderr):
      19:13:07.687 [14981] <2> onlfi_vfms_logf: INF -     clone group 
      and volume already exists
      19:13:07.688 [14981] <2> onlfi_vfms_logf: INF - --- End of file 
      /var/tmp/VfMSBAAr7aOqD

      NetBackup automatically creates VxVM clones for TimeFinder, ShadowImage, or BusinessCopy backups of the data that is configured over volumes. After the backup has completed, NetBackup removes the VxVM clone. In this case, a system crash or restart may have prevented the removal. Remove the clone as follows.

      (Do the following on the client or alternate client, depending on the type of backup.)

    • When no backups are running, use the following VxVM command to list any clones: vxdg list

      The clone name is of the form clone_disk_group_clone.

    • To remove the clone, enter the following:

      /usr/openv/netbackup/bin/bpdgclone -g disk_group -n volume -c

      For example:

      /usr/openv/netbackup/bin/bpdgclone -g wil_test -n vol01 -c

      where wil_test is the name of the disk group and volo1 is the name of the VxVM volume.

      For more information on now to remove a VxVM clone, refer to the NetBackup Snapshot Client Administrator's Guide. For vxdg, refer to the Veritas Volume Manager Administrator's Guide.

    • Before running the backup again, resynchronize the primary disk with the secondary disk. For assistance, refer to the NetBackup Snapshot Client Administrator's Guide.

    • If a snapshot backup fails using the FlashSnap or VVR snapshot method, a VxVM snapshot may be left over from a previous backup. You may see messages similar to the following in the /usr/openv/netbackup/logs/bpfis log:

      14:41:15.345 [22493] <32> onlfi_freeze: FTL - VfMS error 11; see 
      following messages:
      14:41:15.345 [22493] <32> onlfi_freeze: FTL - Fatal method error 
      was reported
      14:41:15.345 [22493] <32> onlfi_freeze: FTL - vfm_freeze_commit: 
      method: FlashSnap, type: FIM, function: FlashSnap_freeze_commit
      14:41:15.345 [22493] <32> onlfi_freeze: FTL - VfMS method error 
      8; see following message:
      14:41:15.345 [22493] <32> onlfi_freeze: FTL - 
      vxvm__find_ready_snapshot: Cannot find available snapshot mirror

      NetBackup automatically creates VxVM snapshots for backups of the data that is configured over volumes. After the backup completes, NetBackup removes the VxVM snapshot. In this case, a system crash or restart may have prevented the removal. Remove the snapshot as follows.

      For FlashSnap:

      (Do the following on the client or alternate client, depending on the type of backup.)

    • Find the VxVM disk group:

            vxdg list

      The format of the disk group name is as follows:

            primaryhost_diskgroup_split

      If vxdg list does not show the disk group, the group might have been deported. You can discover all the disk groups, including deported ones, by entering:

            vxdisk -o alldgs list

      The disk groups that are listed in parentheses are not imported on the local system.

    • Deport the VxVM disk group:

            vxdg deport primaryhost_diskgroup_split

      Enter the following on the primary (original) client:

    • Import and join the VxVM disk group:

            vxdg import primaryhost_diskgroup_split
            vxrecover -g primaryhost_diskgroup_split -m
            vxdg join primaryhost_diskgroup_split diskgroup
    • Start the volume and snap back the snapshot volume:

          vxvol -g primaryhost_diskgroup_split start 
      SNAP_diskgroup_volume
          vxassist snapback SNAP_diskgroup_volume

      For VVR, on the alternate client:

    • Enter the following to display unsynchronized mirror disks:

          vxprint -g diskgroup
    • Enter the following to resynchronize the mirror disks:

            vxassist -g diskgroup -v volume snapback
    • When you use a snapshot method such as VxFS_Checkpoint to back up a Veritas File System (VxFS), the backup fails if the VxFS license has expired. Messages such as the following appear in the /usr/openv/netbackup/logs/bpfis log:

      11:37:42.279 [24194] <2> onlfi_vfms_logf: INF - 
      VxFS_Checkpoint_freeze: Cannot open checkpoint; status=100
      11:37:42.283 [24194] <4> delete_mount_point: INF - Deleted mount 
      point /tmp/_vrts_frzn_img__test1_24194
      11:37:42.283 [24194] <32> onlfi_freeze_fim_fs: FTL - VfMS error 
      11; see following messages:
      11:37:42.283 [24194] <32> onlfi_freeze_fim_fs: FTL - Fatal 
      method error was reported
      11:37:42.284 [24194] <32> onlfi_freeze_fim_fs: FTL - vfm_freeze: 
      method: VxFS_Checkpoint, type: FIM, function: 
      VxFS_Checkpoint_freeze
      11:37:42.284 [24194] <32> onlfi_freeze_fim_fs: FTL - VfMS method 
      error 100; see following message:
      11:37:42.284 [24194] <32> onlfi_freeze_fim_fs: FTL - 
      VxFS_Checkpoint_freeze: Cannot open checkpoint; status=100

      Obtain a new VxFS license and retry the backup.

    • If the backup is enabled for instant recovery with either the VxVM or VVR snapshot method, your VxVM mirrors may not be properly configured. In this case, you may see the following in the /usr/openv/netbackup/logs/bppfi log on the client (when verbose mode is set high).

      13:43:39.095 [16375] <2> onlfi_vfms_logf: INF - Executing 
      command:
      13:43:39.095 [16375] <2> onlfi_vfms_logf: INF - 
      /usr/sbin/vxprint -g rootdg -q -t -e 'assoc="pfi_concat"' 
      </dev/null >/var/tmp/VfMSAA Arja4.F 2>/var/tmp/VfMSBAAsja4.F
      13:43:39.215 [16375] <2> onlfi_vfms_logf: INF - 
      pfi_find_snapdone: 0 SNAPDONE plexes found
      
      13:43:39.215 [16375] <2> onlfi_vfms_logf: INF - Executing 
      command:
      13:43:39.215 [16375] <2> onlfi_vfms_logf: INF - 
      /usr/sbin/vxassist -g rootdg snapprint pfi_concat  </dev/null 
      >/var/tmp/VfMSAAArja4.F 2>/var/tmp/VfMSBAAsja4.F
      13:43:39.512 [16375] <2> onlfi_vfms_logf: INF - 0 active plexes 
      for /rootdg/pfi_concat:  0 are PFI  0 non-PFI
      13:43:39.512 [16375] <2> onlfi_vfms_logf: INF - 
      pfi_find_active.3309: exiting with VXVM_E_SYS = 3
      13:43:39.512 [16375] <2> onlfi_vfms_logf: INF - 
      pfi_snapshot.3866: No PFI snapshot. err= 3

      Configure the VxVM mirrors as described in the Instant Recovery chapter of the NetBackup Snapshot Client Administrator's Guide.

    • When you use the VxFS_Checkpoint snapshot method, the backup fails if the client's file system does not support mountable checkpoints using the Storage Checkpoint feature. Messages such as the following appear in the /usr/openv/netbackup/logs/bpfis log:

      14:54:27.530 [23563] <32> onlfi_freeze_fim_fs: FTL - VfMS error 
      11; see following messages:
      14:54:27.530 [23563] <32> onlfi_freeze_fim_fs: FTL - Fatal 
      method error was reported
      14:54:27.530 [23563] <32> onlfi_freeze_fim_fs: FTL - vfm_freeze: 
      method: VxFS_Checkpoint, type: FIM, function: 
      VxFS_Checkpoint_freeze 14:54:27.531 [23563] <32> 
      onlfi_freeze_fim_fs: FTL - VfMS method error 2; see following 
      message:
      14:54:27.531 [23563] <32> onlfi_freeze_fim_fs: FTL - open_ckpt: 
      Cannot open checkpoint on /mnt_vxvm/2G_concat : 
      fsckpt_get_api_version returns 1; mountable checkpoints not 
      supported with this version

      Do one of the following:

      • Upgrade the client file system to a version that supports mountable VxFS Storage Checkpoints

      • Configure the policy with a snapshot method that supports the client's current file system.


Legacy ID



v15099072_v47163124


Article URL http://www.symantec.com/docs/HOWTO35032


Terms of use for this information are found in Legal Notices