Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

NB Status 41

Created: 01 Jul 2013 • Updated: 06 Jul 2013 | 15 comments
This issue has been solved. See solution.

Hi all,

I am facing a netbackup status code 41.

As of now, I checked the connectivity such a s ping, telnet...all working fine.

If it is related to CLIENT_READ_TIMEOUT, I wil try increasing this.

Please advise.

 

 

Thanks and Regards,

.Jayaram Balasubramanian.

Operating Systems:

Comments 15 CommentsJump to latest comment

Amit Suthar's picture

During a backup, the NetBackup client agent may not be able to send keepalives to the media server if the read of client data is taking a long time to complete one buffer of data. When this occurs, the media server may timeout after the time specified in the media server's host properties.

With database systems, some clients with very large filesystems / active filesystems / lots of small files may need a value of 1800, 3600, 7200, 10800 or higher.

For your system:

Please login to master and/or media server and increase the following timeout to 3600

In the Admin GUI go to Host Properties => Master Server Host Properties => Timeouts => "Client Read Timeout" -> 3600

Also set the "Client Read Timeout" to 3600 for the client on which the backup fails:
Go to Host Properties => Clients => Properties (of the specific client) => Timeouts

They must match or the restore/backup will use the lower value.

After performing the above steps restart the Netbackup services on the Master server so that these settings take effect. If you cannot restart Netbackup services, run :

\netbackup\bin\admincmd\bprdreq -rereadconfig

Try the backup again.

revaroo's picture

Shouldn't need to restart services when adding/increasing the CLIENT_READ_TIMEOUT.

The next new backup should take the parameters.

mph999's picture

STOP.

Unless you have a database, don't increase the client read timeout to 3600 - way too high, and could introduce other issues.

There is not enough information in the original post to give any advice. We need proper details

1. What are you backing up
2. Has this ever worked
3. Is it a new setup
4. Is any data backed up, or does it fail at the beginning
5. What is the current client-read-timeout setting
6. How long after the job starts does the failure happen
7 . Does it always fail after the same amount of time
8. If question 7 = no, and data is backed up, does it always fail after the same amount of data is backed up
9. What OS is the master / media / client
10 . What version(s) of NBU
11. If this previoiusly worked, what changed
12. Do other similar clients work
13. Any firewalls
14. Does the problem only happe at certain times of the day

Sure, it might turn out that the client read timeout needs hanging, perhaps you do have a database backup, and a high setting is justified, but we do not know this yet.

Making assumptions, is the quickest way to totally screw up a system, - ok, one setting is no big deal but wat happens is that multiple settings end up being changed, and then this combination introduces other issues and then as time goes by, it is forgotten what was chaged and you end up with a system that takes a long time to fix.

I'm saying that is the case here, but the issue you have has to be correctly understood before anything is done.

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
Marianne's picture

In addition to above excellent posts, status 41 can be be client read timeout or client connect timeout.

So, in addition to answering all Martin's questions, please post all text in the Job Details.
This will tell us if we are dealing with read or connect timeout.

Please create bpcd and bpbkar log folders on the client for further troubleshooting.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

jayaram226's picture

Hey Martin,

To answer your questions,

1. What are you backing up

It's a File system backup of two directories.

2. Has this ever worked

Master server was down and up for a while.

3. Is it a new setup

No it's not a new setup.

4. Is any data backed up, or does it fail at the beginning

Job doesn't fail immediately.

5. What is the current client-read-timeout setting

900

6. How long after the job starts does the failure happen

it fails after five minutes.

7 . Does it always fail after the same amount of time

Yes.

8. If question 7 = no, and data is backed up, does it always fail after the same amount of data is backed up

9. What OS is the master / media / client

master : Solaris 7 ; Client : Solaris 5.9

10 . What version(s) of NBU

Client : NB 6 MP 4 Master server : NB 6

11. If this previoiusly worked, what changed

No major changes in the host.

12. Do other similar clients work

yes.

13. Any firewalls

Connecitivity is fine. Ping and Telnet works.

14. Does the problem only happe at certain times of the day

No..it happens at all time.

Hope I have given the required answers.

 

Thanks and Regards,

.Jayaram Balasubramanian.

 

Marianne's picture

Please show us all text in Job Details.

We still don't know if this is Connect or Read timeout.

Have you created log folders as per my post above?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mph999's picture

13.  Any firewalls

>> Ping and telnet work.

I am sure they do, but that does not mean there is no firewall, and it does not mean there isn't a firewall closing a port after 5 mins ???

 

M

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
jayaram226's picture

Hi Mariannee,

 

I can't post the job details, as I get in enviroment, but as per your post I created the folders in client.

 

Thanks and Regards,

.Jayaram Balasubramaian.

jayaram226's picture

Hoping this might answer Marianne's question....

Job ends after, A Media is positioned.

bpdbjobs -all_columns "jobid", gives like,

"After position time: - 0 KB Written at 0.000 KB/sec", - end writing."

Thanks and Regards,

.Jayaram Balasubramanian.

Marianne's picture

We need to see ALL information in Details tab.

Important information:
Connecting....
Connected....
etc...

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

jayaram226's picture

Marianne,

I've edited the host inofrmations, and policy informations...I'm just posting the last part of bpdbjobs -all_columns.

 

14:49:01 - started process bpbrm (23407),07/02/13 14:49:06 - connecting,07/02/13 14:49:06 - connected; connect time: 000:00:00,07/02/13 14:49:09 - mounting SB003D,07/02/13 14:51:20 - mounted; mount time: 000:02:11,07/02/13 14:51:26 - positioning "Media id"..to file 1,07/02/13 14:51:42 - begin writing,07/02/13 14:51:47 - positioned; position time: 000:00:21,07/02/13 15:06:47 - 0 KB written - 0.000 KB/sec,07/02/13 15:07:22 - end writing;

 

Thanks and Regards,

.Jayaram Balasubramanian.

 

 

 

revaroo's picture

So nothing is backed up according to 0 KB Written at 0.000 KB/sec

Need to see the job details otherwise we cannot comment on a potential way to fix this 

Marianne's picture

We need to see all of the following logs:

On media server: bptm and bpbrm

On client: bpcd and bpbkar

Please copy log files to .txt files that matches process name (e.g. bptm.txt) and post as File attachments.

 

Edit:

This is what Job details would look like:

14:49:01 - started process bpbrm (23407),
07/02/13 14:49:06 - connecting,
07/02/13 14:49:06 - connected; connect time: 000:00:00,
07/02/13 14:49:09 - mounting SB003D,
07/02/13 14:51:20 - mounted; mount time: 000:02:11,
07/02/13 14:51:26 - positioning "Media id"..to file 1,
07/02/13 14:51:42 - begin writing,
07/02/13 14:51:47 - positioned; position time: 000:00:21,
07/02/13 15:06:47 - 0 KB written - 0.000 KB/sec,
07/02/13 15:07:22 - end writing;
 
 
So, exactly 15 minutes from beginning of backup to timeout.
This is Client Read timeout (900).
We need bpbkar log on the client to see if data stream is generated and network transfer is attempted.
Best to increase Client logging level to something like a 3 or 5 
(bp.conf entry of 
VERBOSE = 5)
 
We need to see bpbrm on media server to see if any file info is received from client.
bptm is needed to see if any data is received from client.
bpcd on client will confirm connectivity and 'handover' to bpbkar.
 
Please check client for any hung bpbkar processes. Your NBU level is very old and was the worst NBU version ever. Support ended last year.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

jayaram226's picture

Hi Marianne,

Backup job is running fine now, and 41 issue was closed.

But still not clear, what that solved the issue.

now I need your help more.

Thanks and Regards,

.Jayaram Balasubramanian.

Marianne's picture

We need the logs for a failed backup that I've mentioned above.

Impossible to know without logs or evidence of hung bpbkar processes at the time.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION