Video Screencast Help

status 84 on MSDP pool, no tape drive attached

Created: 06 Feb 2013 • Updated: 18 Mar 2013 | 10 comments
This issue has been solved. See solution.

Hello.

Master / Media servers - all win server 2008 enterprise r2. Client Red hat 6.2.

I have a client that keeps throwing a status 84. Tried restarting the services (they all shutdown / start up properly), rebooting the media server OS, stil having the same issue. The MSDP pool is not near the high water mark, and the c:\ on the media server has plenty of space.(this has happened on this server before, and these simple steps fixed it - not this time).

 

The disk pool shows as being UP. Is there anything else I can try? As noted above, this media server has no attached tape drive, and is only backing up one client. Any ideas?

 

-Scott

Discussion Filed Under:

Comments 10 CommentsJump to latest comment

Marianne's picture

Client is not causing status 84.

Status 84 is I/O error on media server. 
Please post all text in Details tab of failed job as well as bptm log on media server. 
Please post log as file attachment.

If bptm log folder does not exist on media server, please create it and retry backup. Post resulting log.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

lovethatcheese's picture

Detailed status of job:
_______

2/5/2013 5:44:59 PM - Info bpbrm(pid=1060) apfs1 is the host to backup data from     
2/5/2013 5:44:59 PM - Info bpbrm(pid=1060) reading file list from client        
2/5/2013 5:45:02 PM - Info bpbrm(pid=1060) starting bpbkar32 on client         
2/5/2013 5:45:02 PM - Info bpbkar32(pid=0) Backup started           
2/5/2013 5:45:02 PM - Info bptm(pid=2032) start            
2/5/2013 5:45:02 PM - Info bptm(pid=2032) using 262144 data buffer size        
2/5/2013 5:45:02 PM - Info bptm(pid=2032) setting receive network buffer to 1049600 bytes      
2/5/2013 5:45:02 PM - Info bptm(pid=2032) using 30 data buffers         
2/5/2013 5:45:05 PM - Info bptm(pid=2032) start backup           
2/5/2013 5:45:07 PM - Critical bptm(pid=2032) image open failed: error 2060001: one or more invalid arguments   
2/5/2013 5:45:07 PM - Info bptm(pid=2032) EXITING with status 84 <----------        
2/5/2013 5:45:08 PM - Error bpbrm(pid=1060) from client apfs1: ERR - Cannot write to STDOUT. Errno = 104: Connection reset by peer
2/5/2013 8:45:07 PM - Info nbjm(pid=2092) starting backup job (jobid=1796) for client apfs1, policy apfs1, schedule Differential-Inc  
2/5/2013 8:45:07 PM - Info nbjm(pid=2092) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1796, request id:{70F2E979-605E-46A5-8DB4-05010CF15A92})  
2/5/2013 8:45:07 PM - requesting resource apsu1
2/5/2013 8:45:07 PM - requesting resource nbflmaster.NBU_CLIENT.MAXJOBS.apfs1
2/5/2013 8:45:07 PM - requesting resource nbflmaster.NBU_POLICY.MAXJOBS.apfs1
2/5/2013 8:45:07 PM - granted resource nbflmaster.NBU_CLIENT.MAXJOBS.apfs1
2/5/2013 8:45:07 PM - granted resource nbflmaster.NBU_POLICY.MAXJOBS.apfs1
2/5/2013 8:45:07 PM - granted resource MediaID=@aaaah;DiskVolume=PureDiskVolume;DiskPool=apdp1;Path=PureDiskVolume;StorageServer=apnetbackup;MediaServer=apnetbackup
2/5/2013 8:45:07 PM - granted resource apsu1
2/5/2013 8:45:07 PM - estimated 23624563 Kbytes needed
2/5/2013 8:45:07 PM - Info nbjm(pid=2092) started backup (backupid=apfs1_1360125907) job for client apfs1, policy apfs1, schedule Differential-Inc on storage unit apsu1
2/5/2013 8:45:09 PM - started process bpbrm (1060)
2/5/2013 8:45:10 PM - connecting
2/5/2013 8:45:12 PM - connected; connect time: 00:00:02
2/5/2013 8:45:20 PM - end writing
media write error(84)
2/6/2013 2:36:46 PM - job 1796 was restarted as job 1832

_____

 

AttachmentSize
020513.txt 106.93 KB
020613.txt 39.84 KB

_____

Current Environment - NB 7.5.0.4 on Master / Media Servers. 7+ and above on clients (Red Hat).

OS - Windows Server 2008 R2 Enterprise on all

All media servers have MSDP / attached tape drive / enabled SLP's

___

Dyneshia's picture

Are there any errors in the event viewer during the time of the backup ?  Be sure to look on the media server doing the backup

lovethatcheese's picture

Hello. I hadn't looked at the event viewer yet, but I'm looking at it now, and restarted the job...is there something specific I should be looking in the event viewew in regards to an MSDP pool? May seem like a basic question, but I haven't had too much exposure to the event viewer on windows..

Would this be under Windows logs ?

Applications / Services logs?

 

_____

Current Environment - NB 7.5.0.4 on Master / Media Servers. 7+ and above on clients (Red Hat).

OS - Windows Server 2008 R2 Enterprise on all

All media servers have MSDP / attached tape drive / enabled SLP's

___

Dyneshia's picture

Yes in the application event viewer.  http://technet.microsoft.com/en-us/library/cc76640...

Look for anything with a red X during the time of the backup.

Marianne's picture

What is your NBU 7.5 patch level? There are some MSDP fixes in 7.5.0.4. 

We see errors in bptm log:

 

14:36:46.128 [2380.1836] <2> construct_sts_isid: master_server nbflmaster, client apfs1, backup_time 1360190206, copy_number 1, stream_number 0, fragment_number 0, resume_number 0, spl_name NULL
14:36:46.128 [2380.1836] <2> bp_sts_open_image: STH set to STS_SA_IMAGE
14:37:03.506 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: impl_set_imh_image_prop: PdvfsClose() failed (22:Invalid argument)
14:37:03.506 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: impl_image_handle: impl_set_imh_image_prop() failed (2060001:one or more invalid arguments)
14:37:03.506 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: impl_create_image: impl_image_handle() failed (2060001:one or more invalid arguments)
14:37:03.506 [2380.1836] <32> bp_sts_open_image: sts_create_image failed: error 2060001
....

14:37:03.569 [2380.1836] <32> write_disk_header: image open failed: error 2060001: 
14:37:03.569 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: impl_image_handle: empty image delete success
14:37:03.569 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: impl_delete_image_legacy: could not remove apfs1_1360190206_C1_HDR (2060013:no more entries)
14:37:03.569 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: pi_delete_image_v10: impl_delete_image() failed (2060013:no more entries)
14:37:03.584 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: impl_image_handle: empty image delete success
14:37:03.584 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: impl_delete_image_legacy: could not remove apfs1_1360190206_C1_F1 (2060013:no more entries)
14:37:03.584 [2380.1836] <16> 1832:bptm:2380:apnetbackup: [ERROR] PDSTS: pi_delete_image_v10: impl_delete_image() failed (2060013:no more entries)
14:37:03.584 [2380.1836] <2> delete_image_disk_sts_impl: file apfs1_1360190206_C1_F1 does not exist

Maybe this TN is applicable? http://www.symantec.com/docs/TECH183707

MMM!! Seems we've been here before?? https://www-secure.symantec.com/connect/forums/getting-status-84-client-storage-unit-msdp

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

lovethatcheese's picture

Marianne,

Nice in spotting out my previous post with this client!! - I'll be honest, you are pretty hard to sneak one by.  :)

My master /  media servers are 7.5.0.4 - however, the client is 7.0. The previous fix when opening a case with Symantec (my previous post) didn't work as well this time, so I thought I would see what the forums had to say. I've opened up another ticket, I'll update this thread when I hear from someone.

 

These status 84's sure are something!

_____

Current Environment - NB 7.5.0.4 on Master / Media Servers. 7+ and above on clients (Red Hat).

OS - Windows Server 2008 R2 Enterprise on all

All media servers have MSDP / attached tape drive / enabled SLP's

___

lovethatcheese's picture

Also, as an FYI...this client just went through with an incremental backup with no issues, and I haven't changed a thing....

2/6/2013 1:45:32 PM - Info bpbrm(pid=1184) apfs1 is the host to backup data from     
2/6/2013 1:45:32 PM - Info bpbrm(pid=1184) reading file list from client        
2/6/2013 1:45:34 PM - Info bpbrm(pid=1184) starting bpbkar32 on client         
2/6/2013 1:45:34 PM - Info bpbkar32(pid=0) Backup started           
2/6/2013 1:45:35 PM - Info bptm(pid=1164) start            
2/6/2013 1:45:35 PM - Info bptm(pid=1164) using 262144 data buffer size        
2/6/2013 1:45:35 PM - Info bptm(pid=1164) setting receive network buffer to 1049600 bytes      
2/6/2013 1:45:35 PM - Info bptm(pid=1164) using 30 data buffers         
2/6/2013 1:45:37 PM - Info bptm(pid=1164) start backup           
2/6/2013 1:45:39 PM - Info bptm(pid=1164) backup child process is pid 1172.1964       
2/6/2013 1:45:39 PM - Warning bpbrm(pid=1184) from client apfs1: WRN - Could not reset access time of /home/aquota.user. Errno = 1: Operation not permitted
2/6/2013 1:45:39 PM - Info bptm(pid=1172) start            
2/6/2013 1:53:42 PM - Info bptm(pid=1164) waited for full buffer 146 times, delayed 30388 times    
2/6/2013 1:53:45 PM - Info bptm(pid=1164) EXITING with status 0 <----------        
2/6/2013 1:53:45 PM - Info apnetbackup(pid=1164) StorageServer=PureDisk:apnetbackup; Report=PDDO Stats for (apnetbackup): scanned: 545079 KB, CR sent: 238743 KB, CR sent over FC: 0 KB, dedup: 56.2%
2/6/2013 1:53:45 PM - Info bpbrm(pid=1184) validating image for client apfs1        
2/6/2013 4:45:39 PM - Info nbjm(pid=2092) starting backup job (jobid=1837) for client apfs1, policy apfs1, schedule Differential-Inc  
2/6/2013 4:45:39 PM - Info nbjm(pid=2092) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1837, request id:{EE84E008-51A6-4D52-96C5-F1B885BBA56C})  
2/6/2013 4:45:39 PM - requesting resource apsu1
2/6/2013 4:45:39 PM - requesting resource nbflmaster.NBU_CLIENT.MAXJOBS.apfs1
2/6/2013 4:45:39 PM - requesting resource nbflmaster.NBU_POLICY.MAXJOBS.apfs1
2/6/2013 4:45:39 PM - granted resource nbflmaster.NBU_CLIENT.MAXJOBS.apfs1
2/6/2013 4:45:39 PM - granted resource nbflmaster.NBU_POLICY.MAXJOBS.apfs1
2/6/2013 4:45:39 PM - granted resource MediaID=@aaaah;DiskVolume=PureDiskVolume;DiskPool=apdp1;Path=PureDiskVolume;StorageServer=apnetbackup;MediaServer=apnetbackup
2/6/2013 4:45:39 PM - granted resource apsu1
2/6/2013 4:45:39 PM - estimated 23624563 Kbytes needed
2/6/2013 4:45:39 PM - Info nbjm(pid=2092) started backup (backupid=apfs1_1360197939) job for client apfs1, policy apfs1, schedule Differential-Inc on storage unit apsu1
2/6/2013 4:45:41 PM - started process bpbrm (1184)
2/6/2013 4:45:42 PM - connecting
2/6/2013 4:45:44 PM - connected; connect time: 00:00:02
2/6/2013 4:45:49 PM - begin writing
2/6/2013 4:53:57 PM - end writing; write time: 00:08:08
the requested operation was successfully completed(0)

___

 

 

Is there such a thing as a tempermental media server? :)

 

_____

Current Environment - NB 7.5.0.4 on Master / Media Servers. 7+ and above on clients (Red Hat).

OS - Windows Server 2008 R2 Enterprise on all

All media servers have MSDP / attached tape drive / enabled SLP's

___

Marianne's picture

The opening lines of TECH183707 may be applicable?

NetBackup deduplication backups may fail because the NetBackup Deduplication Engine (spoold) is still loading fingerprints from the database into the cache. This may occur if backup jobs start running before spoold finishes loading the fingerprint cache.

TN also tells how to check...

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
lovethatcheese's picture

Marianne,

Looks like this is the same issue I had before. I have opened up a ticket with Symantec, and will resolve this thread once things are fixed.

 

Per step 5, I am still getting 'Put=no"

___

  1. If "put=Yes," cache loading is complete and normal operations will resume.

    If "put=No," contact your Symantec support representative.

After the initial start of spoold after the upgrade to 7.5, spoold does not read the entire MSDP for fingerprints. Therefore, successive starts of spoold should occur much faster.

___

_____

Current Environment - NB 7.5.0.4 on Master / Media Servers. 7+ and above on clients (Red Hat).

OS - Windows Server 2008 R2 Enterprise on all

All media servers have MSDP / attached tape drive / enabled SLP's

___