Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

the Server returned Error Code 23

Created: 02 Aug 2012 • Updated: 06 Jan 2014 | 28 comments
Zahid.Haseeb's picture
This issue has been solved. See solution.

Environment

Netbackup = 7.1

OS = 2003 R2

Client = Linux

Netbackup Client = 7.1

selinux=disabled

linux client able to resolve the netbackup server

/usr/openv/netbackup/bin/bpclntcmd -ip <client_ip_address>    = successful

 

 

Problem

I am facing the below error while restoring the backup. I also followed the following TN but nothing positive.

 

http://www.symantec.com/business/support/index?pag...

http://www.symantec.com/business/support/index?pag...

 

 

Discussion Filed Under:

Comments 28 CommentsJump to latest comment

Marianne's picture

At which point of the restore do you see this error?
What kind of backup are you trying to restore? Database? Normal Windows policy? Flashbackup?
Can you browse for files/folders?
What happens when you start the restore?
Does it get to Activity Monitor?

Please post bprd and bpdbm logs as attachments.
Tell us what time error was seen and which client you are trying to restore.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

SOrry for late sharing the required logs. Find the attachment for your reference:

AttachmentSize
bpdbm.tar.gz 20.37 KB
bprd.tar.gz 38.05 KB

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

When I initiate restore it gives me this error after a while.

Standard

Yes

In Activity Monitor the Restore job shows in queue.

Its a linux redhat client.

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

On Master/Media Server

 

bplist -l /tmp

EXIT STATUS 227:no entity was found

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

What is the purpose of bplist command?
Without -C client-name, the default is local hostname (master).
On Windows master, default type (-t) is 13 - MS-Windows.
Without start and end date (-s and -e). The search period is last 24 hours.

So, you are trying to list backups done for client 'master' of /tmp?

PS /tmp is not even backed up on Unix/Linux client - automatically excluded.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Kindly see the latest logs for the restore activity which got failed

AttachmentSize
logs.tar.gz 21.85 KB

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

Status Code details from below TN:

http://www.symantec.com/business/support/index?pag...

 

NetBackup status code: 227

Message: no entity was found

Explanation: The item requested was not in the catalog. For example, the entity can be a file or it can be policy information.

Recommended Action: A common cause for this problem is a query that has no matching images. Specify different parameters or options for the operation and try it again.

===========================================================

 ""or it can be policy information.""

What the above highlighted sentence means

Please Note:  I am restoring the images on a different server(not on a server where I took backup from) ...

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

You never told us what the purpose was of bplist command? See my previous explanation of the command that you have typed.

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

appologize : typed wrong

 

(  

On Master/Media Server

 

bplist -l /tmp

EXIT STATUS 227:no entity was found

   )

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Andy Welburn's picture

I agree, it does seem a bit vague to say the least, but if you do not get the policy type correct, for example, in a bplist command then despite there being backups present that conform to all the other options you have supplied then you'll get an EC=227.

Marianne's picture

What exactly are we looking for in the new set of logs?

I can see in bprd log that you started browsing for images for client 192.168.1.102 at 12:15:

12:15:12.593 [1444.2204] <2> process_request: imagelist request = 329199 81 192.168.1.102 tpsbkp.tpsonline.com tpsbkp NONE 0  12 0 1376378112 en en en en en 11 0 0 0 0 0 0 0

Restore was started at 12:23:

12:23:57.606 [5412.4480] <2> process_msgQ_msg: Got new restore request from pid 4496

 

12:23:57.606 [5412.4480] <2> read_needed_images_from_file: reading C:\Program Files\Veritas\NetBackup\bin\bprd.d\neededimage.4496 for work to do
 
 
Media server and media id identified: 
 
12:23:57.606 [5412.4480] <2> add_to_tape_list: A00023 tpsbkp
 
Restore started on master/media server:
 
12:24:33.312 [5412.4480] <2> read_bpbrm_msg: (pid 4172) CURRENT POSITION A00023 2
12:24:33.312 [5412.4480] <2> send_bpbrm_msg: (pid 4172) CONTINUE RESTORE
12:24:34.413 [5412.4480] <2> read_bpbrm_msg: (pid 4172) CKP - 10642.001 1344817359 0 0 6 1 0 Z 22 /home/ali.ahmed/.zshrc
12:24:35.013 [5412.4480] <2> read_bpbrm_msg: (pid 4172) EXIT 192.168.1.102_1341312277 0
12:24:35.013 [5412.4480] <2> handle_image_status: 192.168.1.102_1341312277 restfiles pid 4496 bpbrm pid 4172 status = 0
12:24:35.013 [5412.4480] <2> remove_completed_restore: unlink record with restfiles pid 4496
12:24:35.013 [5412.4480] <2> remove_completed_restore: no more active restores
12:24:35.027 [5412.4480] <2> done_with_bpbrm: bpbrm pid 4172 has no more images to process
12:24:35.027 [5412.4480] <2> send_bpbrm_msg: (pid 4172) TERMINATE
12:24:35.070 [4496.4532] <2> restorefiles: Done with port 0 backup_id 192.168.1.102_1341312277. Status = 0
12:24:35.099 [4496.4532] <2> mail_msg_and_set_exit_status: RESTORE EXIT STATUS = 0
 
The above restore was successful.
 
According to bprd log, another restore was submitted at 12:25.
Last entry in bprd log is at 12:39.
No error.
 
 

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

This line is also present under bprd logs.

12:25:12.120 [6036.1152] <2> get_type_of_client_free_browse: db_getCLIENT() failed: no entity was found (227)

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

One more thing I would like to add here. In the response of your above explanation. I tried to restore a selected backup ( its actually a very tiny directory ). It got successful (may be showed in the provided logs)

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

That status 227 is perfectly normal. You will see that the code is <2> which is notice/debug info.
<8> is warning, <16> is error, <32> is severe error. So, no error.

The status 227 is for  'get_type_of_client_free_browse' . This means that you have not added this client to Master Server's Client Attributes with specific browse/restore permissions for this client.

We still have not any seen any logs for the status 23?

All we can see is logs for successful restore.

Your original screenshot shows error in submitting restore request. Was this a once-off?
What other processes were running on the master at the time?
We often see bprd timeout on 32-bit masters (even on 64-bit masters) when large restore requests are submitted, for example large amount of files/folders from FlashBackup.
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

when large restore requests are submitted

The above words are making me more confident. Yes its a very large no. of files and folders. I should have increase the timout value more as per the above TN I mentioned.

http://www.symantec.com/business/support/index?page=content&id=TECH38975

Let me do it and will share the result.

 

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Zahid.Haseeb's picture

Unfortunately I tried it but could not successful. One more thing I have noticed. I tried to restore the folders one by one and it got successful except three folders which made the restore activity hangged. The three folders I did not restore. See the below for reference:

sudo du -hs /home/xxx_data/ /home/xxxxnix/ /home/admin
32G    /home/xxx_data/
18G    /home/xxxxnix/
7.8G    /home/admin

Second If I try to restore these three folders I see the below screen


 

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

I did not have much hope that 'user-directed operation timeout ' in TECH38975 would solve the problem.

I have come to know this as a problem with bprd memory utilization, especially on 32-bit master servers.

The only solution/workaround that I have found is to select smaller sets of data.

Please see this discussion: https://www-secure.symantec.com/connect/forums/status-restore-image-created-mon-may-21-160143-2012-allocation-failed#comment-7172871

What you are seeing above is that bprd is hanging. 
Only way out is to stop NetBackup Request Manager, then kill all bprd processes (check in Task Manager that all are indeed killed). Restart Request Manager.
Restarting Request Manager will only affect restores and user backups.

If you choose to restart NBU, verify that there are no bprd processes in Task Manager after stopping NBU.
Hung bprd processes may not terminate when NBU is stopped - kill them manually.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

thanks for the workaround.

  • Means yet no proper solution ?
  • Is not any automated restore which itself take a small chunk of files and did restore and when that chunk finished it again get next chunk of files for restore.

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

I remember seeing a fix for this in some 7.5.x documentation. Will see if I can find it.

There is no way within NBU to automate restores. You can try to script it.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Thanks Marianne. Me waiting :)

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

I see that I have mentioned it in the post that I referred to above - 

 

... documented in NBU 7.5 EEB Guide:

The bprd process runs out of memory when it prepares restores with millions of files.

Link in NBU 7.5 Documentation: http://www.symantec.com/docs/DOC5138 :

NetBackup 7.5 Emergency Engineering Binary Guide 

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Upgraded the Netbackup Master/Media Server only but still same problem.

( Should upgrade client to 7.5 ? )

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

Hopefully you have upgraded to 7.5.0.3 before image migration took place? 

See this note in Upgrade Portal

 

4. NetBackup 7.5 Maintenance Release 3 (7.5.0.3) should be applied immediately after the 7.5 installation has been completed.  NetBackup 7.5.0.3 download information can be found at the following link:
 

Have you verified that the image migration was completed before you attempted another restore?

 
5. NetBackup 7.5 migrates the NetBackup catalog image header information into the Netbackup relational database. This is a multistep process that may impact installation and operational performance until complete. Please review the following PDF for more details:
 

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Ouch !!!! did not installed the patch...The link you provided has three files which are below :

NB_CLT_7.5.0.3-tar-split.1of3

NB_CLT_7.5.0.3-tar-split.2of3

NB_CLT_7.5.0.3-tar-split.3of3

 

Just want to make sure that these files are for windows Netbackup 7.5 version ?

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

Those are not the only links on that page. *CLT* patches are for Unix/Linux clients.

You probably want Windows patches?
Extract from http://www.symantec.com/docs/TECH189607

WINDOWS:  

Note:  Windows packages include fixes for Add-on products and Database Agent.

NB_7.5.0.3.winnt.x86.exe provides fixes for NetBackup 7.5 on 32-bit Windows server and clients.
www.symantec.com/docs/TECH189617

NB_7.5.0.3.winnt.x64.exe provides fixes for NetBackup (tm) 7.5 on x64 Windows server and clients.
www.symantec.com/docs/TECH189616

 

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
Zahid.Haseeb's picture

upgraded to 7.5.0.3 tried to restore but did not successful...Now seeing what you have said :

5. NetBackup 7.5 migrates the NetBackup catalog image header information into the Netbackup relational database. This is a multistep process that may impact installation and operational performance until complete. Please review the following PDF for more details:

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com

Marianne's picture

If the problem persists, please log a Support call with Symatec. 

Refer them to eTrack 2411170 in NetBackup_75_EEB_Guide.pdf and tell them that you are still experiencing the same proble.

Please be prepared to supply LOTS of level 5 logs (although 1 should be sufficient).

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Zahid.Haseeb's picture

Yes Marianne the same thing I will do and thanks for your kind responses. I really appriciate your help :)

Any comment will be appreciated. Mark as Solution if your query is resolved
__________________
Thanks in Advance
Zahid Haseeb

zahidhaseeb.wordpress.com