Video Screencast Help

Client backup failing with error 25

Created: 04 Jan 2013 • Updated: 03 Jun 2013 | 47 comments
anishC5's picture
This issue has been solved. See solution.

Hi All,

We had done an extensive analysis for the last 2 months on this issue.

Client Server OS: win svr 2008 64 bit

checked the BPCD log with the following error:

08:08:44.891 [34140.41260] <4> bpcd valid_server: server1.backup.local is not a master server
08:08:44.891 [34140.41260] <16> bpcd valid_server: server1.backup.local is not a media server either

We do have the master and media server entires done on the Netbackup properties of the client.

error code 46: server access denied.

 

I tried to delete the total content from registry but after deleting all it is holding onto the master server entry.

Is there anything we can do or logs we can check please?

 

Thanks,

Anish

Comments 47 CommentsJump to latest comment

anishC5's picture

hi Ranjanna,

The TN was good and as checked we have everything in place.

Please can u suggest that the client server is holding onto the master server entry even when I delete it from registry.

Any ideas please where this entry could be?

Thanks,

Anish

Marianne's picture

What EXACTLY do you have under client's Server entries in the registry?

Use 'regedt32' (not regedit), double-click on Server - it will bring up a text-based window that you can copy.

Please post entries here.

Please note that NBU is case sensitive and entries must match reverse lookup exactly (server1.backup.local).

Server1.backup.local != SERVER1.backup.local != server1 != server1.backup.local
 
If any changes to DNS, hosts files and/or registry was made, please remember to clear NBU host cache on server and client:
bpclntcmd -clear_host_cache
 

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi marianne,

please find the entries:

nbu-not-clust

nbu-not-node1

nbu-not-node2

nbu-not-med1

nbu-not-med2

From where do I check the reverse lookup please?

Marianne's picture

You check reverse lookup from the client with:

bpclntcmd -ip <server-ip>

This command uses OS system call 'gethostbyaddr' to resolve IP to hostname (see http://www.symantec.com/docs/TECH27430 )

You can see in client's bpcd log file that it is resolving the server's incoming IP address as server1.backup.local.

You can also see that this server name does not exist in Server entries on the client.

Add this hostname to Server entries (copy exact name from client's bpcd log & paste in registry entry.
No need to battle with this for 2 more months!

If you feel that reverse lookup is incorrect, check name resolution at OS level - hosts file as well as DNS.
Remember to clear NBU host cache if any changes are made...
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

thanks for your suggestions.

Cleared the host cache after adding teh server entry to the "Server" tab in registry.

restarted the services on client.

still having failures.

anyhting I have missed out please?

EDIT: i will arrange to restart the server once and then give u the results.

Thanks,

Anish

mph999's picture

From the media  run nbdna script  on netbackup/bin/support

nbdna -phase 2

Do the same from the client - grab the results

I'm with Marianne, some name resolution issue somewhere I suspect ....

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
anishC5's picture

Hi Martin,

The backup failed again and BPCD gave the same error code 46.

We have around 3000 + servers in our environment.

Is there any filter we can use in nbdna command only for this particular client please from the media server?

Thanks

Anish

 

Marianne's picture

When Server entry matches EXACTLY the reverse lookup hostname that you see in client's bpcd log, your error will disappear.

Please go through the TN in my previous post to verify hostname lookup.

No need to restart client.

If you don't feel comfortable sharing real entries in a public forum, please copy actual bpcd log as well as Server entries in the registry into a private message to one of us trying to help you.

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Marianne's picture

I am looking at your other post: https://www-secure.symantec.com/connect/forums/user-initiated-sql-backups-are-failing-230-though-policy-and-schedule-are-place

If I read through the thread again, it seems that you have a backup network.

Important to ensure that Public and Backup NICs have different hostnames.

Add entries on local hosts files and also ensure that DNS is updated with separate hostnames for each IP address.

Sample hosts entry on media server (lets say 192.168 is the public network and 10.10 the backup network):

192.168.10.1 media1
10.10.10.1 media1-bck

Client:
192.168.10.2 client1
10.10.10.2 client1-bck

Ensure that client1-bck is used in Clients tab of policy and that client has Server entry for media1-bck.

PS: PLEASE update your other post as well?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi Marianne,

 

i have shared the BPCD and bpclntcmd command outputs with u in a message.

What I feel is from client to media and master server their are 2 entries coming up.

COuld that make a differnece. i have tallied with some other clients and this is not the case.

 

Thanks,

Anish

anishC5's picture

hi marianne, martin.

i think I have found the issue. Please rectify me if wrong.
While comparing the BPCD log with a successful client I found the following anomalies.
Please can you guide me as to where do I make the necessary changes.

unsuccessful client:

11:04:54.961 [32252.38356] <2> logconnections: BPCD ACCEPT FROM 172.30.210.51.54176 TO 172.30.210.51.13782 fd = 732
11:04:54.962 [32252.38356] <2> process_requests: setup_sockopts complete
11:04:54.967 [32252.38356] <2> bpcd peer_hostname: Connection from host Client-server02.backup.local (172.30.210.51) port 54176
11:04:54.967 [32252.38356] <2> bpcd valid_server: comparing master-server-clus and Client-server02.backup.local
11:04:54.968 [32252.38356] <2> bpcd valid_server: comparing master-server1 and Client-server02.backup.local
11:04:54.969 [32252.38356] <2> bpcd valid_server: comparing master-server2 and Client-server02.backup.local
11:04:54.969 [32252.38356] <2> bpcd valid_server: comparing med-server651 and Client-server02.backup.local
11:04:54.970 [32252.38356] <2> bpcd valid_server: comparing med-server652 and Client-server02.backup.local
11:04:54.970 [32252.38356] <2> bpcd valid_server: comparing med-server653 and Client-server02.backup.local
11:04:54.971 [32252.38356] <2> bpcd valid_server: comparing med-server654 and Client-server02.backup.local
11:04:54.971 [32252.38356] <2> bpcd valid_server: comparing med-server655 and Client-server02.backup.local
11:04:54.972 [32252.38356] <2> bpcd valid_server: comparing med-server656 and Client-server02.backup.local
11:04:54.972 [32252.38356] <2> bpcd valid_server: comparing med-server657 and Client-server02.backup.local
11:04:54.973 [32252.38356] <2> bpcd valid_server: comparing med-server658 and Client-server02.backup.local
11:04:54.974 [32252.38356] <2> bpcd valid_server: comparing med-server659 and Client-server02.backup.local
11:04:54.974 [32252.38356] <2> bpcd valid_server: comparing master-server.backup.local and Client-server02.backup.local
11:04:54.975 [32252.38356] <2> bpcd valid_server: comparing Client-server02.backup.local and Client-server02.backup.local
11:04:54.975 [32252.38356] <4> bpcd valid_server: hostname comparison succeeded

successful client:

14:43:08.983 [20108.30156] <2> logconnections: BPCD ACCEPT FROM 172.30.240.1.1946 TO 172.30.216.214.1556 fd = 696
14:43:08.983 [20108.30156] <2> process_requests: setup_sockopts complete
14:43:08.989 [20108.30156] <2> bpcd peer_hostname: Connection from host master-server-clus.backup.local (172.30.240.1) port 1946
14:43:08.989 [20108.30156] <2> bpcd valid_server: comparing master-server-clus.backup.local and master-server-clus.backup.local
14:43:08.989 [20108.30156] <4> bpcd valid_server: hostname comparison succeeded

Mark_Solutions's picture

Both of those do show a successful connection and validation but one is from a client and one from the master server

Maybe you have made a type in its hosts file or similar so although the comparison successds it is resolved incorrectly and does not reply to the correct server.

Clear down the logs and try a connection to each from the Master server again and then show us the two logs

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

anishC5's picture

hi mark,

 

Sorry but I did not follow you. Which logs do you want me to clear please?

Do you want me to clear out the BPCD log or any log on the master server please?

Since I have been strugling with this problem for over 2 months now so my brain is gone into hang mode

Sorry, Anish

 

Marianne's picture

Your opening post shows a failed connection, not the last post...

The one that you describe as unsuccessful is a connection from the client itself. Is this a SAN media server backing up itself? 

I have just had a look at the bpcd log that you sent via PM.

The actual problem is not with host lookup as in the bpcd log in opening post. Seems problematic socket connection back to itself.

bpcd entries after successful hostname comparison:

11:04:55.230 [33780.23280] <4> bpcd valid_server: hostname comparison succeeded
11:04:55.231 [36048.35092] <2> process_requests: output socket port number = 2466
11:04:55.231 [33780.23280] <16> process_requests: read failed: The operation completed successfully.
11:04:55.236 [36048.35092] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1527: 0: in failed file cache: 10109 0x0000277d
11:04:55.236 [36048.35092] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1528: 0: in failed file cache name: NULL
11:04:55.236 [36048.35092] <2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1529: 0: in failed file cache service: testdaemon
11:04:55.236 [36048.35092] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2485: 0: vnet_cached_getaddrinfo failed: 6 0x00000006
11:04:55.236 [36048.35092] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2486: 0: vnet_cached_getaddrinfo failed ret: 10109 0x0000277d
11:04:55.236 [36048.35092] <2> vnet_cached_get_service_port: ../../libvlibs/vnet_addrinfo.c.2487: 0: vnet_cached_getaddrinfo failed service: testdaemon
11:04:55.236 [36048.35092] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1827: 0: vnet_cached_get_service_port() failed: 6 0x00000006

11:04:55.236 [36048.35092] <2> is_pbxable_server: ../../libvlibs/vnet_connect.c.1828: 0: service: 2466

11:04:55.238 [36048.35092] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6646: 0: fopen() failed: 2 0x00000002
11:04:55.238 [36048.35092] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6647: 0: fopen() failed: C:\Program Files\Veritas\NetBackup\var\host_cache\04f\1ef2244f+2466,1,0,2,1,0+172.30.210.51.txt
11:04:55.246 [36048.35092] <2> process_requests: <---- NetBackup 7.1 0 ------------initiated
11:04:55.246 [36048.35092] <2> process_requests: VERBOSE = 0
11:04:55.246 [36048.35092] <2> process_requests: Not using VxSS authentication with Client-server02.backup.local
11:04:55.246 [36048.35092] <2> process_requests: BPCD_OPEN_FOR_READ_RQST
11:04:55.246 [36048.35092] <2> process_requests: open for read file name = C:\Program Files\Veritas\NetBackup\var\LoginBanner.conf
11:04:55.246 [36048.35092] <2> process_requests: buffer length = 32766
11:04:55.246 [36048.35092] <2> process_requests: our open flags = 0x0
11:04:55.246 [36048.35092] <2> process_requests: client open flags = 0x8000
11:04:55.246 [36048.35092] <16> process_requests: couldn't open file for read. errno = 2: No such file or directory
11:04:55.246 [36048.35092] <2> process_requests: BPCD_DISCONNECT_RQST
11:04:55.246 [36048.35092] <2> bpcd exit_bpcd: exit status 0 ----------->exiting
11:05:09.421 [5692.5696] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied. )

 

Lets have another look at the connection last post request in the post above:

logconnections: BPCD ACCEPT FROM 172.30.210.51.54176 TO 172.30.210.51.13782 fd = 732

WHY connecting to bpcd port??
Why output socket port number 2466?
 

Connect to and connect back default port is 1556 for NBU 7.1.

Successful client:
BPCD ACCEPT FROM 172.30.240.1.1946 TO 172.30.216.214.1556

Correct port 1556 is connected to.

Has anybody changed any settings for this client (san media server?) in Host Properties?

Check Host Properties -> Master -> Client Attributes -> Client-name  -> Connect Options. 
All 3 fields should say Use default connect options.

Also check that Windows Firewall is disabled and that there is no other software preventing socket setup.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

yes marianne you are right. the socket port should be 1566 and not 2466.

i have chekd in the master server properties for the client-server and all 3 tabs in connect options is set to "use default'

Any changes we need to make on the client to alter this value please?

Thanks,

Anish

Marianne's picture

Can you confirm that this has worked previously?
Or is this a new config that has never worked before?
Can you confirm that this is indeed a SAN Media Server?

Have you confirmed that Windows Firewall is disabled on this machine?

Have you checked for third-party software that could be blocking or preventing port connection and/or socket setup?
Some Anti-Virus software is doing this. We have even seen that Download Manager sofware was preventing socket setup.
Some other ideas as to what third-part software to check for:
http://www.symantec.com/docs/TECH88628

We don't see TCP/IP 10038 error in your bpcd log, but worth a look.

We do see TCP/IP 10022 error in your log. I have Googled the error which points to misconfigured sofware or even 'improperly-formatted IPv4 or IPv6 Network Address'.

Have you tried to remove and reinstall NBU software?

You have not replied to my question about miltiple NICs.
Have you confirmed that each NIC/IP address has a different hostname?

You really need to figure out what exactly is different on this server. 

Some other things to check: 

Compare contents of C:\Windows\System32\drivers\etc\services file with other (working) systems.

Host Properties -> Media Server: Select this server -> Firewall
Compare with working media server. Ensure all settings are the same.

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

Thanks marianne.

I will follow ur suggestions and reply to all ur queries at once. May take some time but I will.

Thanks,

Anish

anishC5's picture

hi Marianne,

i am answering few of your queries. rest I am working with server and network to sort out:

 

Can you confirm that this has worked previously?- yes, it has worked proviously
Or is this a new config that has never worked before?
Can you confirm that this is indeed a SAN Media Server?- this is just a client server. We have seperate media servers(our media servers are all SAN media servers)

Have you confirmed that Windows Firewall is disabled on this machine?- yes it is disabled

Have you checked for third-party software that could be blocking or preventing port connection and/or socket setup?
Some Anti-Virus software is doing this. We have even seen that Download Manager sofware was preventing socket setup.
Some other ideas as to what third-part software to check for:
http://www.symantec.com/docs/TECH88628
-Checking for this one. this is a citrix svr but another citrix server is working fine.

We don't see TCP/IP 10038 error in your bpcd log, but worth a look.

We do see TCP/IP 10022 error in your log. I have Googled the error which points to misconfigured sofware or even 'improperly-formatted IPv4 or IPv6 Network Address'.- I will check with network team once regarding the network setup.

Have you tried to remove and reinstall NBU software?- yes(but i will have a talk to our server guys if it was repaired or reinstalled, probably reinstalled only)

You have not replied to my question about miltiple NICs.
Have you confirmed that each NIC/IP address has a different hostname?- yes

You really need to figure out what exactly is different on this server.

Some other things to check:

Compare contents of C:\Windows\System32\drivers\etc\services file with other (working) systems.-no difference found ( no mention of port 1556 and 2466)

Host Properties -> Media Server: Select this server -> Firewall
Compare with working media server. Ensure all settings are the same.

-All settings found same for all media servers.

 

Thanks,

Anish

Marianne's picture

Can you confirm that this is indeed a SAN Media Server?
- this is just a client server. We have seperate media servers(our media servers are all SAN media servers)

Does Client Server mean a normal NBU client?

We see in bpcd log that incoming connection request is coming from the SAME IP address as the client: 

unsuccessful client:

11:04:54.961 [32252.38356] <2> logconnections: BPCD ACCEPT FROM 172.30.210.51.54176 TO 172.30.210.51.13782 fd = 732

Why do we see the connection request coming from the SAME IP address? 
Possibly duplicate IP addresses for media server and this client?
But if that was the case, we would not see bpcd entry on the client as media server would actually connect to itself!

In the successful client you can see incoming connection request from a different IP address which indicates that the connection request is coming from a media server:

logconnections: BPCD ACCEPT FROM 172.30.240.1.1946 TO 172.30.216.214.1556 fd = 696

Please check the following on the Media server backing up this client, on the client and the master (all in the TN that I've posted above):

On media server:
bpclntcmd -self 
bpclntcmd -hn <client-name> 
bpclntcmd -ip <client-IP>

On client:
bpclntcmd -self 
bpclntcmd -hn <media-server-name> 
bpclntcmd -ip <media-server-IP>

On master:
bpclntcmd -hn <client-name> 
bpclntcmd -ip <client-IP>
bpclntcmd -hn <media-server-name> 
bpclntcmd -ip <media-server-IP>

 

(You can send output in a PM if you don't want to share real hostnames).
 

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

Hi Marianne,

i understand that things might be a bit confusing in this case.

ideally it should be getting the BPCD accept from master  / media server to client not client to client.

We have had got the NBu uninstalled and reinstalled.

Not sure if the reinstallation was done wrongly or if sumthing was changed.

We are in touch with the server team for the same. Network seems alright as per network team.

ping, telnet etc are all ok from all sides.

Thanks, Anish

Marianne's picture

You need to find out where this connection request from same IP is coming from.

When was bpcd log collected - directly after failed backup attempt?
Or after connection test such as bptestbpcd?

Thanks for the PM - I can see that hostnames are resolving fine from Master, media server and client.

Have you tried to rename bpcd log (e.g Client010813.log, to ensure a new log is created) and then do bptestbpcd from master server?
bptestbpcd -client <client-name> -debug -verbose
Check the result of the command, then check result in client's bpcd log.

Rename bpcd log again (e.g Master-Client010813.log) to ensure a new log is created.
Repeat bptestbpcd test from media server.
Check the result of the command, then check result in client's bpcd log.

If all of the above checks out, you need to double-check policy config - Storage unit that is selected (plus STU properties) as well as client name in Clients tab. 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi Marianne,

i am running the command from master server but is stuck at:

08:08:58.668 [9240] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.445: 0: VN_REQUEST_CONNECT_FORWARD_SOCKET: 10 0x0000000a
08:08:58.677 [9240] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.462: 0: ipc_string: 54270
 

Is this a warning or error or some kind please?

So far there has been no update of the bpcd log after renaming

Thanks,

Anish

Marianne's picture

If connection cannot be made, it will sit there until Client Connect Timeout kicks in.

Leave it until timeout occurs.
The final error messages should be useful. You should be able to see source and destination IP addresses  and most important, the destination ports that NBU tries to connect to.

Please show us full output when error occurs.

If you have done Ctrl-C in the meantime, check Client Connect Timeout. If it is more than 300 (5 minutes), drop it down to 300 so that timeout and error will occur within 5 minutes.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi Marianne,

master server settings are taking some time to load for observation.

Ill let you know the output soon.

What is most frustrating is, I know what the error is. I know that when we are running the backup for this client, the server is being detected as the master server and is communicating on a different port. But what I dont know is where do I change the settings.

:(

Marianne's picture

Then you have done something wrong when you updated Client's registry entries.

Ensure Master is first in the list, then normal media servers following master's entry.

Remove itself from Server list. We initially thought this was a San Media server.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

Thanks Marianne,

Will do that. i think the settings are the same since I have not changed anything.

Will it help if I send u the reg settings in a PM please from both successful and non-successful client?

Thanks,

Anish

Marianne's picture

Apologies for only responding now... I was quite busy today.

Yes, please send settings in PM. It will be good to compare working client with non-working client.

Not sure where in the world you are - it is quite late over here. I will check for reply/PM in the morning. 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi Marianne, we are based in INDIA. i have created both the txt files . just waiting for server team to transfer them to me from the clients.. successful and unsuccessful.

Mny thanks for all ur help on this case. hopefully we will be able to resolve this issue.

Marianne's picture

Certainly hope we can get it fixed today....

Have you allowed bptestbpcd to finish so that we can see error messages and see if anything is logged in client's bpcd log?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi Marianne,

 

I have sent you the reg files.

i could not finish the bptestbpcd since it is hanging in between and putty session is getting disconnected.

Ill try to run from a different server to see the results. I will share the observation soon.

Thanks,

Anish

Marianne's picture

Reduce Client Connect Timeout before you try bptestbpcd. 

You should be able to increase Putty's timeout as well.

The "Client" key for the problematic client contains rows and rows of hex code.
I have checked this key on my laptop - it contains my local hostname only.

Compare Server keys between working and non-working clients (the PM did not contain Server entries).

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

yes Marianne,

You are right.  I tried to download them in text format but they keep getting downloaded in HEX format only.

i ll comapre them and try to send you the results along with bptest bpcd results.

Thanks,

Anish

anishC5's picture

hi Marianne,

 

yes please i will try to convince server team to uninstall NBU, remove registry and then reinstall the same giving example of registry of a successful client as well.

Somehow I feel that a proper reinstallation will fix the issue.

i have sent you the BPCD output of both master and media servers.

error code 23.

Thanks,

Anish

Marianne's picture

Thanks for the PM with bptestbpcd output.
I have not checked media server attempt yet, but we see the following from master:

First connection attempt on PBX (1556):

BPCD CONNECT FROM <master-ip>.14823 TO <client-ip>.1556 fd = 3 

do_vnetd_service: ../../libvlibs/vnet_connect.c.1641: 0: remote host supports PBX, but PBX is not running: 0 0x00000000

Next, vnetd connection is attempted (13724):

do_vnetd_service: ../../libvlibs/vnet_connect.c.1675: 0: connect: VNETD CONNECT FROM <master-ip>.19024 TO <client-ip>.13724 fd = 4

10:39:27.587 [22938] <2> vnet_vnetd_connect_forward_socket_begin: ../../libvlibs/vnet_vnetd.c.462: 0: ipc_string: 53733

10:43:09.524 [22345] <2> get_short: (1) cannot read (byte 1) from network: Interrupted system call (4)
10:43:09.525 [22345] <2> bpcr_put_vnetd_forward_socket: get_short failed: 4
10:43:09.525 [22345] <2> local_bpcr_connect: bpcr_put_vnetd_forward_socket failed: 23
10:43:09.525 [22345] <2> ConnectToBPCD: bpcd_connect_and_verify(<client-name>, <client-name>) failed: 23
<16>bptestbpcd main: Function ConnectToBPCD(<client-name>) failed: 23
10:43:09.525 [22345] <16> bptestbpcd main: Function ConnectToBPCD(<client-name>) failed: 23
<2>bptestbpcd: socket read failed
10:43:09.525 [22345] <2> bptestbpcd: socket read failed
<2>bptestbpcd: EXIT status = 23

Was anything logged in Client's bpcd?

Can you tell us which NBU services are running on the Client?

The following services should be running:

NetBackup Legacy Client Service
NetBackup Client Service
NetBackup Legacy Network Service
Symantec Private Branch Exchange

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi we have entries in BPCD but since the verbose was 5 as asked byt symantec engineer, so there are lot of entries. I'll share the detials in a PM.

I have had a talk to my line manager and we are going with a complte reinstall of the NBU.

Also all the 4 services are running.

I'll let you all know the results.

Thanks,

Anish

Marianne's picture

Any chance client is behind firewall with NAT enabled?

See this TN: http://www.symantec.com/docs/TECH66968

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

Hi marianne,

there are no firewall in the backup network.

Thanks,

Anish

Marianne's picture

This is getting very confusing...

How many NICs on this client?

We see this IP address in all of the output supplied up to now (even 'bpclntcmd -self'):
172.30.210.51

Verbose version of bpcd log that you sent via PM shows 2 IP addresses:
10:19:21.634 [49200.48036] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.132: pbxRegisterEx successful at 172.30.210.51:5126/bpcd, returns with 1 alt_addrs
10:19:21.634 [49200.48036] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.144: alt_addr: 10.215.237.139:5126

bptestbpcd was attempted from master between 10:37 and 10:43, but there is no record in bpcd of any connection attempt. 

bptestbpcd was attempted from Media server between 11:07 and 11:13, but once again, no record in bpcd of any connection attempt. 

And yet you say that telnet is working? On which port?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi Marianne,

Apologies for al lthe confusion.

For every client in our environment, we have 2 network NICs minimum.

1. corporate NIC: starting with either 192 or 10

2. backup NIC: starting with 172

 

I will send you the telnet results from master and media server in a while over PM.

I am telnetting the client over ports 13782, 13724 and 1556

Thanks,

Anish

Marianne's picture

Mysteries of the world... telnet to all ports from master and media server is fine.
And we still have no idea where initial connection from itself to itself was coming from as no tests that we have done over the last couple of days could replicate the problem. bptestbpcd tests are not reflected in client's bpcd log.

I wonder if 'something' in the registry got corrupted during this process:

I tried to delete the total content from registry but after deleting all it is holding onto the master server entry.

The multiple rows of hex code in Clients key does not seem right. We cannot say what else is wrong.

My suggestion is still the same: 
Before any more troubleshooting is done, uninstall NBU software. 
Reboot Client
Check that all NBU registry entries have been removed and that there are no left-over Services entries.
(There is a NBU 6.0 TN on how to cleanup registry after failed upgrade that can be used.)
Reboot again if registry had to be cleaned up.
Install & patch NBU Client software, taking care that only valid Server entries are added during installation.
Create bpcd log folder.
Do bptestbpcd from master and media server.

Let us know the result...

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

Yes please. Very strange indeed.

Anyways i have initiated the request and is upto our server team to carry out the installation, reboot etc.

I will let you know the results whenever I have them,.

Thanks for all your help marianne.

Cheers, Anish

anishC5's picture

hi,

Just an update. We have begun the uninstallation, reboot, removal of registry enties for the problematic client we were discussing here.

Uninstallation and reboot is done.

Thanks, Anish 

anishC5's picture

hi Marianne,

We have uninstalled and reinstalled the NBU but it is still failing with the previous errors.

However when I compared the reg keys this time with both successful and unsuccessful client having lesser keys, i observed a difference.

In unsuccessful client inside "CONFIG" there is a entry called:

Connect Options: Local Host 1 0 2

whereas in successful client there is no such entry. I am going to get that entry deleted from unsuccessful client and check again.

is it worth please?

Thanks,

Anish

Marianne's picture

Which NBU version did you install on Client?

My own NBU installation on my laptop (NBU 7.5) shows:
localhost 0 0 2

Did you remember to create bpcd log on client?
Is anything logged in bpcd log after failed backup and/or connection attempt?
If so, please share.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

anishC5's picture

hi,

Wee the only update available to me regarding this is that there is no update. backups are still failing. and we have asked the server team to build a server with same config etc from scratch and install NBu to check if that works.

We aksed them to rebuild the server in VM keeping the current one alive.

let us see if that works. Unfortuantely being a production server we cannot run too many trail and error on it.

Ill see how long that takes etc.

Thanks,

Anish

anishC5's picture

hi All,

I am happy to share the fact that we had finally managed to resolve this issue.

We had problem trying to access the client via the VNETD port, so with help from symantec backline engineer we bypassed it. and directly connecting via BPCD port.

the backups are now successful

Thanks.

SOLUTION