Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Server Status: Communication with the server has not been initiated or the server status has not been retrieved

Created: 07 Mar 2013 • Updated: 27 Mar 2013 | 16 comments
This issue has been solved. See solution.

Hi,

For the past few hours, for multiple Oracle/RMAN backups across multiple Unix/Linux servers, the backups are failing with the following RMAN error stack:

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch00 channel at 03/07/2013 14:08:04
ORA-19506: failed to create sequential file, name="bk_1367_1_809444249", parms=""
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
   VxBSACreateObject: Failed with error:
   Server Status:  Communication with the server has not been initiated or the server status has not been retrieved from the serve
 
I suspect ther might be some kind of network problem between the NetBackup master and the database servers and would value any insights anyone out there might have on this.
 
Thanks!
Allan
Operating Systems:

Comments 16 CommentsJump to latest comment

AllanSeabrook's picture

Thanks, WR. Actually, I did find Marianne's post earlier today, but was hoping for a few more tips on what to look for. I suspect we might have a network issue as these backups were working without problem until yesterday afternoon and nothing has changed on the database servers.

Marianne's picture

As per the other post, you need bprd log on the master and dbclient on the client.
You can also test comms from the client with 'bpclntcmd -pn'.

We saw a similar issue a day or two ago when someone added another client name in master's /etc/hosts file using same IP address.
Master was now resolving Client's incoming IP address as incorrect hostname.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

AllanSeabrook's picture

Thanks for your suggestions. Could you perhaps help me locate the bprd log on the master (Windows server) and dbclient on the client (Unix and Linux servers)? There are multiple Unix/Linux servers affected, but the NetBackup software setup on each of them seems to be the same:

/usr/openv/netbackup

Attached are two log files from /usr/openv/netbackup/logs/user_ops/dbext/logs from one of the Linux servers, one from before the problem started, the other during the problem.

29854.0.1362462941_successful.txt

13692.0.1362700065_failed.txt

An edited NetBackup/RMAN output log from one of the Linux boxes is also attached showing successful as well as failed backups.

NetBackupRMAN.txt

I appreciate any suggestions you might have.

Allan

AttachmentSize
29854.0.1362462941_successful.txt 1.49 KB
13692.0.1362700065_failed.txt 1.11 KB
NetBackupRMAN.txt 14.3 KB
Marianne's picture

Will have a look at the attachments just now...

About logs: 
They don't exist by default and must be created.

bprd on Master:
Create bprd folder under <install-path>\netbackup\logs followed by restart of NBU Request Service.

dbclient on Oracle client:
mkdir /usr/openv/netbackup/logs/dbclient
chmod 777 /usr/openv/netbackup/logs/dbclient 

*** EDIT ***

Looks like 5-minute timeout:

18:47:46 Initiating backup
18:52:00 INF - .....

Without logs, I'm not sure if this is a Timeout on the master or the media server. Probably media server.

It won't do any harm to check and increase Timeouts on both (if master and media server are different).
Change Client Connect and Client Read Timeout to 1800.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

AllanSeabrook's picture

Thanks Marianne,

We have now created the log directories on both the Master and client and restarted the NBU Request Service. Both logs now contain entries that will probably agree with your timeout recommendation.

Attached are:

bprd_2013-03-08.txt - from D:\Program Files\Veritas\NetBackup\logs\bprd on NetBackup Master

dbclient_2013-03-08.txt - from /usr/openv/netbackup/logs/dbclient on Linux database server

NetBackupRMAN_2013-03-08.txt - the latest log showing the RMAN failure.
 
Interestingly, since we added the log directories and restarted the NBU Request Service, the RMAN error messages have changed.
 
Thanks,
Allan
AttachmentSize
bprd_2013-03-08.txt 21.02 KB
dbclient_2013-03-08.txt 5.55 KB
NetBackupRMAN_2013-03-08.txt 2.16 KB
Marianne's picture

Found this in bprd log:

<32> bprd: Error occurred during initialization.  Could not read logging configuration file.

and this recent post: https://www-secure.symantec.com/connect/forums/error-occurred-during-initialization-could-not-read-logging-configuration-file-0

In this case /etc/vx/vrtslog.conf was missing on the client. 

Another possibility is corrupt nblog.conf.

Check /usr/openv/netbackup/nblog.conf on client as well as <install-path>\Veritas\NetBackup\nblog.conf

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
AllanSeabrook's picture

Hi Marianne,

You were 100% correct! The nblog.conf file was found to be corrupt. Still waiting for details on which one our infrastructure partner corrected since the one on one of our Oracle servers was recently updated as was the one on the master server. 

Thanks again for sharing your knowledge with this forum. It is much appreciated!

Allan

Marianne's picture

Curious to know which nblog.conf.

I would start by checking timestamp on the master.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

AllanSeabrook's picture

Hi Marianne,

Apologies for the delay in providing an update. It turned out that the nblog.conf on the master had become corrupt. Once it was rebuilt, the backups started running normally again.

Such a simple fix, but the problem sure brought our backups to a screeching halt!

Interestingly, I'm now dealing with the same error condition with Oracle backups on a new development Linux server. The Oracle databases on the matching production server have no backup issues, only the dev server. The two boxes were setup in very much the same way at the same time.

I'm going to backtrack through this thread to see if anything jumps out at me. Will keep you posted.

Many thanks,
Allan

AllanSeabrook's picture

Hi Marianne,

You have no idea how helpful you have been! For my latest problem with the Oracle backups on the new Linux server, as per your earlier recommendation, I created the following directory...

mkdir /usr/openv/netbackup/logs/dbclient
chmod 777 /usr/openv/netbackup/logs/dbclient

And the backups on that server now are working!

Thank you so much!

Allan

Marianne's picture

LOL! dbclient log is supposed to assist with troubleshooting, not solve the problem!

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

AllanSeabrook's picture

Yep, the latest failure was being caused by a missing log directory. I must have missed a step during the NetBackup installation on the Linux client.

Marianne's picture

Allan, I have a slighty different theory about the latest problem... Successful backup after creating the log folder may be pure coincidence...

When new clients are added and DNS/hosts files are updated on servers and clients, NBU does not immediately consult these updated entries.

NBU host cache was introduced as a 'feature' in version 7.0.1, where NBU will refresh cache only once every hour. That is why backups that fail initially seem to magically fix itself after a while.

Maybe add a note to refresh NBU host cache on master and media servers when new DNS or hosts entries are added:
bpclntcmd -clear_host_cache
See http://www.symantec.com/docs/TECH136792

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

AllanSeabrook's picture

Hi Marianne,

I think you are on to something. My linking the creation of the missing log directory and the backups starting to work is probably just be a 'red herring'. When the server was commissioned at the end of last year and the databases and NetBackup backup jobs put in place, the backups always failed. So, I disabled them and resorted to disk backups until I found some time to research why the NetBackups for the databases on this server were failing.

Yesterday, on getting back to the failed backups (which hadn't run since the end of last year), it dawned on me that the error reported was the same as that during that multi-day outage we had earlier this month and that's what caused me to return to this thread, create the missing log directory and sucessfully rerun the backups.

Do your think if might have been simply a matter of timing? Or that nblog.conf corruption perhaps being around longer than previously thought?

Perhaps we'll never know, but at least I'm very happy right now!

Thanks again,

Allan

Marianne's picture

I guess we will never know... LOL!

Seems nblog.conf is easily corrupted by disk-full condition or unexpected master server crash/shutdown.

My own nblog.conf nightmare after disk-full on master server:
 https://www-secure.symantec.com/connect/forums/slps-stopped-processing

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links