Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Restores and bpverify from MSDP fails with error code 83

Created: 29 Jan 2013 • Updated: 21 Feb 2013 | 11 comments
mohsinmansoor's picture
This issue has been solved. See solution.

Hi,

I have configured an MSDP pool on a media server residing on IBM AIX. Details are:

Netbackup 7.5.0.4 (One Master Server, One Media Server on AIX and one Media server on Windows)

IBM AIX 6.1 TL6

When i run a backup on the MSDP created on AIX based Media Server it show successful, however if i try to restore or verify the backup it gives me media open error. It was working fine since last Friday,but suddenly it started giving me errors. The only thing that was changes was the creation of additional SLP policies on this MSDP, which are currently disabled. and i dont think that creation of SLP policies should have any affect.

Below are the outputs from the jobs failing.

MEDIA VERIFY output:

01/29/2013 16:30:53 - Info bpdm (pid=8519922) requesting nbjm for media
01/29/2013 16:30:55 - Critical bpdm (pid=8519922) get image properties failed: error 2060013: no more entries
01/29/2013 16:30:57 - Critical bpdm (pid=8519922) image open failed: error 2060018: file not found
01/29/2013 16:30:57 - Error bpbrm (pid=8454238) from client MEDIASERVER: no data in archive
01/29/2013 16:30:57 - Info tar (pid=6357158) done. status: 6
01/29/2013 16:30:57 - Info tar (pid=6357158) done. status: 83: media open error
no images were successfully processed (191)

RESTORE output:

01/29/2013 14:54:43 - Error bptm (pid=6357218) more than 10 files were not restored, remaining ones are shown in the progress log.
01/29/2013 14:54:43 - Info tar (pid=19005492) done. status: 83: media open error
01/29/2013 14:54:43 - Error bpbrm (pid=5570566) client restore EXIT STATUS 83: media open error
media open error (83)

Also i had Storage Lfecycle policies configured on this MSDP which are failing, which i think is also because of this media open error, since the SLP is unable to read the image from the msdp.

DUPLICATION job output: (duplication to Tape)

01/29/2013 12:06:29 - Error bpduplicate (pid=9306244) host MEDIASERVER backup id xxxxxxxxxxx_1359461002 read failed, media open error (83).
01/29/2013 12:06:29 - Error bpduplicate (pid=9306244) host MEDIASERVER backupid xxxxxxx_1359461002 write failed, media manager killed by signal (82).
01/29/2013 12:06:29 - Error bpduplicate (pid=9306244) Duplicate of backupid xxxxxxxxxxxxxxxx_1359461002 failed, media manager killed by signal (82).
01/29/2013 12:06:29 - Error bpduplicate (pid=9306244) Status = no images were successfully processed.
01/29/2013 12:06:29 - end Duplicate; elapsed time 0:00:13
01/29/2013 12:06:33 - Critical bpdm (pid=9568464) image open failed: error 2060018: file not found
01/29/2013 12:06:33 - Error bptm (pid=8650828) media manager terminated during mount of media id 7468L5, possible media mount timeout
01/29/2013 12:06:35 - Error bptm (pid=8650828) media manager terminated by parent process
no images were successfully processed (191)

REPLICATION job output:

01/29/2013 12:06:18 - granted resource MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=PDC-MSDP-1;Path=PureDiskVolume;StorageServer=MEDIASERVERxxxx;MediaServer=MEDIASVRxxxxx
01/29/2013 12:06:24 - Error nbreplicate (pid=18546746) ReplicationJob::Replicate: Replication failed for backup id xxxxxxxxxxxxxx_1359461002: media open error (83)
01/29/2013 12:06:24 - Replicate failed for backup id xxxxxxxxxxxxxxxxx_1359461002 with status 83
01/29/2013 12:06:25 - Info bpdm (pid=5570562) started
01/29/2013 12:06:25 - started process bpdm (pid=5570562)
01/29/2013 12:06:31 - Critical bpdm (pid=5570562) get image properties failed: error 2060013: no more entries
media open error (83)

Any Comments, Suggestions, Resolution would be really appreciated.

Comments 11 CommentsJump to latest comment

Nicolai's picture

My 2 cent of advice. Whenever you get media related errors on MSDP devices open a support ticket with Symantec. Especially when they show up out of nowhere.

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

RLeon's picture

Just an uninformed guess, what is the stoarge that you are using for the MSDP deduplication pool?

iSCSI slower than 10Gb is not supported.
CIFS or NFS share is not supported.

The currently supported storages for MSDP are FC SAN storage and direct-attached-storage (DAS, i.e., the server's own harddrives).

mohsinmansoor's picture

Hi RLeon, we are using FC SAN storage, and the backups were running fine until last week and they have suddenly stopped.

Agreed Nicolai, yes, I have opened a case with Symantec Support, but sometimes i see more experienced responses on Symantec Connect so i posted a query here as well.

mohsinmansoor's picture

In our environment, we also followed the following technote when we were doing the deployment. http://www.symantec.com/business/support/index?pag... I have verified the links in the specified path, but its still failing.

Mark_Solutions's picture

Has anything changed with DNS?

Odd question I know but when using MSDP you find that even though all servers use short host names the pd.conf often uses the FQDN

Backups are fine but duplictations / replications use the FQDN

Host file entries can often sort this - just check that each media server can resolve itself (and other media severs) by their FQDN - take a look at the pd.conf on the Media Servers to see what its FQDN is set to as far as MSDP is concerned and make sure that is resolvable using DNS or hosts file.

The only other thing is the good old case sensitive issue ... in the MSDP storage under \databases\catalog\2\ appears the clients names. If a client appears twice in lower and upper case (or any mix of case) then restores, verifies, duplications and replications can fail.

It may have only just cropped up if you added a new policy for the client and it was listed in a different case.

To resolve it you just need a softlink betweem the two directories for that client

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

mohsinmansoor's picture

Hi Mark,
Thanks for your response. I have verified the entries in /etc/hosts and they are same as we have on DNS server. Also the FQDN is resolvable from the Master and Media server (for itself and Master Server).

However i am wondering about your suggestion to check pd.conf. I cant find any entry pointing to the hostname of MEDIA server or MASTER in pd.conf files.
The paths that i am looking at for pd.conf are

/usr/openv/lib/ost-plugins/pd.conf
/usr/openv/pdde/pdag/pd.conf

WE have also symlinks created on the mini catalog path.

watsons's picture

What about the backups writing to the windows media server, does it have MSDP and if so, did it work?

If that's working, the issue would be MSDP on the AIX media server. 

Few technotes (in 7.5 late breaking news) that may help: 

http://www.symantec.com/docs/TECH183707

http://www.symantec.com/docs/TECH190087

Verify this with support.

mohsinmansoor's picture

Hi Watsons, Thanks for your reply.
we have MSDP on windows based media server and its working fine for backups and restores.

For the technotes you provided,

1st technote states problems in backup and for error code 84, however we are having problem in restores and media verification with error code 83. However i still executed the command /usr/openv/pdde/pdcr/bin/crcontrol --getmode and status was PUT=YES which is correct as per the technote.

the second technote states the resolution is to move to ver 7.5.0.4 and we are already on 7.5.0.4

Mark_Solutions's picture

the pd.conf i referred to was the /usr/openv/lib/ost-plugins one - the ones i have seen have what look like web links to puredisk but use the FQDN of the media server

The crcontrol --getmode is an idea though ... what is the full output of that command on your system?

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

I De Pedro's picture

Hello,

Have you checked this TN?

http://www.symantec.com/docs/TECH152331

 
I know that it's for Windows but I have experienced a similar issue in one UNIX MSDP.
 
Hope this helps.
mohsinmansoor's picture

Hi guys, after struggling a lot on this issue and working with Symantec support. The final descision was to reinstall the netbackup servers. The issue was fixed but after reinstallation. I know its not the right way to resolve, but this is what Symantec Support recommended.

SOLUTION