Video Screencast Help

Netbackup- Status Code 48: the DB backups suddenly failed

Created: 08 Jan 2013 • Updated: 21 Jan 2013 | 8 comments
This issue has been solved. See solution.

Hi All,

I just enountered strange problem, some of the DB backup suddely got problem, the db backup consist of MS_SQL and oracle backup using perl script. The backup policies are not new, they are running fine for quite long.  The error status code is 48. The funny thing the file system on the same server still running fine. I know this is something to do with the name resolution(?) but I have checked with the nslookup for both client and master/media server, looks ok.

What could be possible problem? Any idea how to resolve it?

 

(Master server is running on HPUX 11.11 with Netbackup enterprise server 6.5.6)

 

Thank you very much.

 

Iwan

Discussion Filed Under:

Comments 8 CommentsJump to latest comment

Marianne's picture

MS_SQL and oracle backup using perl script.

A bit more info, please...

Are DB clients running backups to disk followed by bpbackup to backup the disk backups to NBU?

Or Agent backups started by external scheduler on the clients?

Please provide all text in Details tab of failed job.

Check at OS level on master and media server that forward and reverse lookup is correct between server(s) and client(s).

Maybe DNS and/or hosts files have changed?

You will need logging to troubleshoot the problem.
If user-initiated backups, bprd log on master server is first to check.
Next log is bpbrm on the media server(s).

PS: Are you aware that NBU 6.5 support ended in Oct last year?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
Nagalla's picture

hi 

DO you have hosts file or DNS in place for the name resolutions?

if the if the resolution query too more time it may be possible for getting EC 48?

if you are using DNS its more likely possible.

please add the Client entires in etc/hosts file and check how it works

Iwan Tamimi's picture

Thanks Marianne for the quick response

>Are DB clients running backups to disk followed by bpbackup to backup the disk backups to NBU?Or >Agent backups started by external scheduler on the clients?

To make it easier maybe I concentrate on the MS-SQL agent backup first ignore the oracle. Yes our backup started by external scheduler but on the master server

>Please provide all text in Details tab of failed job.

Jan 8, 2013 4:28:56 PM - Error bprd (pid=20182) Unable to write progress log </C/ProgramÀ Files/Veritas/NetBackup/Logs/user_ops/dbext/logs/5180.0.1357633728> on client bccprj01-bck. Policy=BCCPRJ01_DB_BCCPRJSP_MSDB_HOT Sched=NONE
Jan 8, 2013 4:28:57 PM - Error bprd (pid=20182) CLIENT bccprj01-bck  POLICY BCCPRJ01_DB_BCCPRJSP_MSDB_HOT  SCHED NONE  EXIT STATUS 48 (client hostname could not be found)
 

>Check at OS level on master and media server that forward and reverse lookup is correct between server(s) and client(s).

Checked, looks fine

> Maybe DNS and/or hosts files have changed?

Yes there is a possibility, because the policies that failed are not new , just out of sudden it failed, but I don't know what to checked.

>You will need logging to troubleshoot the problem.
>If user-initiated backups, bprd log on master server is first to check.
>Next log is bpbrm on the media server(s).

I will check on it

> PS: Are you aware that NBU 6.5 support ended in Oct last year?

Hehe yes, that's why I am asking here, we are in the middle of upgrading.

 

Thanks Marrianne

Nagalla's picture

Jan 8, 2013 4:28:57 PM - Error bprd (pid=20182) CLIENT bccprj01-bck  POLICY BCCPRJ01_DB_BCCPRJSP_MSDB_HOT  SCHED NONE  EXIT STATUS 48 (client hostname could not be found)

bprd is not finding the host name... 

Please provide the output of below commands in master server

bpcltncmd -hn bccprj01-bck

bpclntcmd -Ip <IP of bccprj01-bck>

grep bccprj01 /etc/hosts

nslookup bccprj01-bck

 

Iwan Tamimi's picture

Hi Nagalla,

 

 

ebs1-bck@/> bpclntcmd -hn bccprj01-bck
host bccprj01-bck: bccprj01-bck at  xx.xx.81.199 (0xa9551c7)
aliases:
ebs1-bck@/> bpclntcmd -ip xx.xx.81.199
checkhaddr: host   : bccprj01-bck: bccprj01-bck at xx.xx.81.199 (0xa9551c7)
checkhaddr: aliases:
ebs1-bck@/> grep bccprj01 /etc/hosts
xx.xx.81.199   bccprj01-bck
ebs1-bck@/> nslookup bccprj01-bck
Using /etc/hosts on:  ebs1-bck
 
looking up FILES
Name:    bccprj01-bck
Address:  xx.xx.81.199
 
 
(notes: I mask our network, but it gave a consistence result)
 
Thank Nagalla.
 
Iwan 
Nagalla's picture

please provide the log of bprd for the process 20182

if the verbose is not 5, please raise up the verbose to 5 and let the job fail then collect the logs of bprd.

and also, how many hosts entires you have in /etc/hosts file?

if it has more entiries, could you move this client entires to beginning of the /etc/hosts fine and check how it working

Iwan Tamimi's picture

Hi All,

Yesterday morning after midnight the problem resolved by itself , I didn't really fix it, I still don't know what wase the caused but here wassome facts:

o The day the thing happened there aws a problem in one of our exchange server, I suspect it could be our secondary DNS server for Windows(The Windows DNS, the domain like company.local, our other DNS is in l Linux for serving company.com.sg), this Windows DNS manage by other department.
o The problem only affecting the one that use SQL Agent and SAP agent.
o During the problem I did nslookup or ping with the name from Master, Media and client to each other and were OK.
o For the client the name resolving done on ..\etc\hosts (but I didn't check wheter it is also automatically registed on the Windows DNS)

Anyway thank you very much for Marianne and Nagala for the supports.

Regards,

Iwan

I will still monitor the situation if I found something I will post it to this forum

Iwan Tamimi's picture

I found something else, it could be the caused, at the same day I also edit the /etc/hosts I accidently put a note without a remark (#) then at night I did restore back the /etc/hosts.

 

Regards,

 

Iwan