Video Screencast Help

Offline catalog backup fails with error 124

Created: 05 Feb 2014 | 21 comments

Hi All,

 

Offline catalog backup is continously failing since 31st Jan in our netbackup environment, below is error which we are receiveing at the end of catalog backup. Please advice. 

 

/6/2014 12:17:53 AM - begin Catalog Backup
2/6/2014 1:14:02 AM - end Catalog Backup; elapsed time: 00:56:09
NB database backup failed, a path was not found or is inaccessible(124)
2/6/2014 1:14:02 AM - Error bpbackupdb(pid=9532) Offline catalog backup to path \\server\Apps\veritascatalog FAILED      
Operating Systems:

Comments 21 CommentsJump to latest comment

Mark_Solutions's picture

offline catalog backup!! That implies you are still on NetBackup 6.x - please upgrade as soon as possible it has been end of life for some time!

As for you error you need to confirm that the path  \\server\Apps\veritascatalog is actually accessible using the account specified in the offline catalog backup settings (right click on the Catalog in th Admin Console and select configure catalog backup - or similar, it has been a while since I did an offline catalog backup!)

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

ramkr2020's picture

hi Mark,

 

path is accessible from master/media server, i've make sure the nb services are using the domain account which have access to the path. but still this error. it was working fine till 31st Jan. suddenly this problem started. not sure what happened. is there a way to find it out? Configure Netbackup catalog screenshot attached. 

Capture.PNG
revaroo's picture

What happens if you goto the path on the master and try and write into it using OS commands?

ramkr2020's picture

attached the screenshot, where it write the image on the path and then it fails.. so not sure why it's failing. 

Capture2.PNG
SymTerry's picture

What version of NetBackup are you running?

TECH49110 talks about this error being caused by a lock file in place. STREAMS.lck files are to do with the NetBackup's stream discovery process - for multiple streams backups (and some other type of backups, too) NetBackup will initiate stream discovery process, and the lock file would be created. While this was for version 5 it might still apply

Do you have a STREAMSpolicy_name.lck in the netbackup/db/images/client/xxxxxxx directory?

 

ramkr2020's picture

hi

nb version is 6.0, i know it's older we have plan to upgrade it soon...yes i can find the streams.lck file in the path you mentioned, i have went throught the article earlier but i couldn't follow the below step.. i am not sure what 3rd pathy tool they mention here and how to remove the lock on the file. 

Customer might use a third-party utility to find which process/application is holding the file handle, and remove the lock on the file to delete it and run a successful catalog backup.

Marianne's picture

Simply delete the .lck file.

PLEASE upgrade this environment ASAP. Not only has all versions of 6.x reached EOSL, it is a known fact that 6.0 was the most problematic version of NBU.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

ramkr2020's picture

Hi Marianne,

 

i agree it's a older version but unfortunately we couldn't upgrade it immediately, but will do it sooner or later. there are .lck file in all servers that are getting backedup. we have master/media server as same, backing up few other servers.. not sure which .lck file i should delete. 

 

 

capture3.PNG
Marianne's picture

Delete all the .lck files in the folder for the master server hostname.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

ramkr2020's picture

hi Marianne,

Deleted the .lck files in master/media server and triggered the offline catalog backup again. Will let you know whether it completes successfully. Thank you all for helping me out with diff ideas!

Mark_Solutions's picture

Also check the amount of disk space on that location

I can see files being writtent there so it works to a certain point before failing - perhaps that location is just almost full so the catalog is too big to write it there

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

ramkr2020's picture

Mark, i can confirm space is not an issue since server have lot of free space. 

Mark_Solutions's picture

Just noticed that the job fails after about an hour

Is the location you are backing up to behind a firewall?

It could be that the keep alive timeout on the firewall (or you operating system) is hitting its 1 hour limit causing the backup to fail

If it is behind a firewall get your firewall admin to increase the keep alive timeout for you

What O/S is your Master Server?

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

ramkr2020's picture

.lck file deletion didn't fix the issue. is not running behind a firewall.. Master Server OS is windows server 2003 R2. 

ramkr2020's picture

Is there any other suggestions on this catalog backup failure. with error: 124. Appreciate your help.

Marianne's picture
  1. Upgrade this environment as a matter of urgency.
  2. Backup to local storage (disk or tape)
  3. Configure Hot Catalog backup

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

ramkr2020's picture

Thanks Marianne, Hot catalog backup is configured, it's getting completed with few errors. but failing the save the fails the Disaster Recovery path /sending emails.... does have access to the path. Is there any settings to check it.

Mark_Solutions's picture

In relation to the original error it could even be a timeout - check the Client Read timeout on the Master and Media Servers host properties - increase it to 3600

In relation to the DR file not being saved it depends where you are saving it - the path needs to be somewhere that the NetBackup client service account has access to write to - what path have you specified?

If it also has some errors what are they - is it just the path or is it more? - show us the text from the detail tabs of the three catalog backup jobs that run

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Marianne's picture

Please ensure that you have logging enabled on the master server.

Important log folders needed to troubleshoot:
bpbkar and bpdbm
(NBU needs to be restarted to enable bpdbm log)

Please copy logs to .txt files to reflect process names (e.g. bpbkar.txt) and upload as File attachments.

MOST IMPORTANT:

UPGRADE your NBU environment to a supported version!

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

ramkr2020's picture

Below is the Error: on the hotling catalog backup logs is

 

2/11/2014 5:03:45 AM - begin writing

2/11/2014 5:03:45 AM - positioned L442L4; position time: 00:00:02

2/11/2014 5:41:35 AM - Error bpbrm(pid=20996) from client apsydbu01: ERR - failure reading file: C:\Program Files\VERITAS\NetBackup\db\images\apsydnd01\1213000000\NT-Servers_1213181070_FULL.f (WIN32 1: Incorrect function. )
2/11/2014 5:48:13 AM - end writing; write time: 00:44:28
the requested operation was partially successful(1)
 
The job was successfully completed, but some files may have been
busy or unaccessible. See the problems report or the client's logs for more details.
 
Offline catalog backup activity logs:
2/11/2014 1:42:03 AM - begin Catalog Backup
2/11/2014 2:37:59 AM - end Catalog Backup; elapsed time: 00:55:56
NB database backup failed, a path was not found or is inaccessible(124)
2/11/2014 2:37:59 AM - Error bpbackupdb(pid=20584) Offline catalog backup to path \\apsydfs01\Apps\veritascatalog FAILED      
 
Attached the bpbar and bpdm log of master/media server. 
capture6.png
AttachmentSize
backuplogs.zip 453.12 KB
Marianne's picture

Seems problem starts with this image file:

1:42:07.579 AM: [22388.35072] <4> backup_create: ......
2:27:23.166 AM: [22388.35072] <2> tar_base::V_vTarMsgW: WRN - error reading file: C:\Program Files\VERITAS\NetBackup\db\images\apsydnd01\1213000000\NT-Servers_1213181070_FULL.f (WIN32 1: Incorrect function. )
2:27:23.166 AM: [22388.35072] <2> tar_base::V_vTarMsgW: WRN - error reading byte: 305152
2:27:23.166 AM: [22388.35072] <2> tar_base::V_vTarMsgW: WRN - wanted buffer size: 32768
2:27:23.322 AM: [22388.35072] <16> dtcp_write: TCP - failure: send socket (1832) (TCP 10054: Connection reset by peer)

Seems this is an image file dating back to 2008:
UNIX time 1213181070 is 06/11/2008 10:44am

At the same time, it also seems as if the network connection got broken:

dtcp_write: TCP - failure: send socket (1832) (TCP 10054: Connection reset by peer)
dtcp_write: TCP - failure: attempted to send 172 bytes

Suggestions from my side:

1. Check database consistency with 'bpdbm -consistency 2 move' (See http://www.symantec.com/docs/HOWTO43651 )

2. As a test, find a USB hard drive, connect it to the master, configure it as Disk storage unit, change catalog backup to this local drive. If this works, you need to find out what is wrong with the network share (not NBU issue...)

3. UPGRADE your environment!

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links