Video Screencast Help

Backup failing with status 58-bptestbpcd failing from mediaserver with status 25

Created: 29 Oct 2012 | 9 comments

Hi,

Master and media:netbackup 7.5 on RHEL

Client:Windows2008R2

Backups for client is failing with status 58,pings,nslookup,telnet,bpclntcmd,bptestbpcd are working fine from MASTER.pings,telnet,bpclntcmd,nslookup working fine but only bptestbpcd is failing with status 25 on MEDIASERVER(on MASTER everything is success).I tried to created bpcd dir on client and run bptestbpcd but there was no log created on client side.Please help.

 

<16>bptestbpcd main: Function ConnectToBPCD(xxxxx_bk.cernerasp.com) failed: 25
cannot connect on socket
 

We have 2 networks one is primary and other is _bk,when I run backups on primary it works fine,so I was suspecting on _bk network problem,but how can I prove it to network/unix guys as pings,nslookup and telnet are working fine only bptestbpcd is failing that too on MEDIAserver and they will ask what is bptestbpcd,how can you say that its network issue when bptestbpcd fails?

Discussion Filed Under:

Comments 9 CommentsJump to latest comment

Marianne's picture

Have you updated hosts files or DNS recently?

If so, have you refreshed NBU host cache on media server?

bpclntcmd -clear_host_cache

Is there a firewall between media server and client?
If so, have backup-IPs been added to firewall rules?

Please show us output of all the following:

On Media server:
bpclntcmd -hn <client_bk-name>
bpclntcmd -ip <client_bk-IP>

On Client:
bpclntcmd -hn <media_bk-name> 
bpclntcmd -ip <media_bk-IP>

On Media server:
bptestbpcd -client <client_bk-name> -debug -verbose

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Naseer Shaik's picture

 

Have you updated hosts files or DNS recently?

I think we have DNS servers,hosts entries are not reuired,however I added all entries to client,media and master

If so, have you refreshed NBU host cache on media server?

bpclntcmd -clear_host_cache

Done

Is there a firewall between media server and client?
If so, have backup-IPs been added to firewall rules?

Yes there is firewall,not sure how to check whether backup-IPs been added to firewall rules or not but I have added bpcd and vnetd to allow

Please show us output of all the following:

On Media server:
bpclntcmd -hn <client_bk-name>
bpclntcmd -ip <client_bk-IP>

 

07:15:03 # bpclntcmd -hn cerncdisql101_bk.cernerasp.com
host cerncdisql101_bk.cernerasp.com: cerncdisql101_bk.cernerasp.com at 10.159.144.189
aliases:     cerncdisql101_bk.cernerasp.com     10.159.144.189
root@taspmonbj02:/etc
07:16:21 # bpclntcmd -hn 10.159.144.189
host 10.159.144.189: cerncdisql101_bk.cernerasp.com at 10.159.144.189
aliases:     cerncdisql101_bk.cernerasp.com     10.159.144.189
 

On Client:
bpclntcmd -hn <media_bk-name> 
bpclntcmd -ip <media_bk-IP>

C:\Program Files\VERITAS\NetBackup\bin>cd admincmd
The system cannot find the path specified.

C:\Program Files\VERITAS\NetBackup\bin>bpclntcmd -hn taspmonbj02_bk.cernerasp.co
m
host taspmonbj02_bk.cernerasp.com: taspmonbj02_bk.cernerasp.com at 10.159.128.17

aliases:     taspmonbj02_bk.cernerasp.com     10.159.128.17

C:\Program Files\VERITAS\NetBackup\bin>bpclntcmd -hn 10.159.128.17
host 10.159.128.17: taspmonbj02_bk.cernerasp.com at 10.159.128.17
aliases:     taspmonbj02_bk.cernerasp.com     10.159.128.17

C:\Program Files\VERITAS\NetBackup\bin>

On Media server:
bptestbpcd -client <client_bk-name> -debug -verbose

07:06:30 #  bptestbpcd -client cerncdisql101_bk.cernerasp.com -debug -verbose
07:09:16.852 [15835] <2> bptestbpcd: VERBOSE = 0
07:09:16.852 [15835] <2> ConnectionCache::connectAndCache: Acquiring new connection for host taspmonbj01.cernerasp.com, query type 223
07:09:16.854 [15835] <2> vnet_pbxConnect: pbxConnectEx Succeeded
07:09:16.854 [15835] <2> logconnections: BPDBM CONNECT FROM 170.71.1.193.15465 TO 170.71.1.192.1556 fd = 3
07:09:16.854 [15835] <8> vnet_check_vxss_client_magic_with_info: [vnet_vxss_helper.c:871] Ignoring VxSS authentication 2 0x2
07:09:16.859 [15835] <2> db_CLIENTsend: reset client protocol version from 0 to 8
07:09:16.900 [15835] <2> db_end: Need to collect reply
07:09:16.900 [15835] <2> db_freeEXDB_INFO: ?
07:12:25.903 [15835] <8> async_connect: [vnet_connect.c:1653] getsockopt SO_ERROR returned 110 0x6e
07:12:37.903 [15835] <8> async_connect: [vnet_connect.c:1653] getsockopt SO_ERROR returned 110 0x6e
07:12:49.903 [15835] <8> async_connect: [vnet_connect.c:1653] getsockopt SO_ERROR returned 110 0x6e
07:14:17.905 [15835] <8> async_connect: [vnet_connect.c:1219] ran out of time before connect 301 0x12d
07:14:17.905 [15835] <8> async_connect: [vnet_connect.c:1219] ran out of time before connect 301 0x12d
07:14:17.905 [15835] <8> async_connect: [vnet_connect.c:1219] ran out of time before connect 301 0x12d
07:14:17.905 [15835] <16> connect_to_service: connect failed STATUS (18) CONNECT_FAILED
        status: FAILED, (44) CONNECT_TIMEOUT; system: (115) Operation now in progress; FROM 170.71.1.193 TO cerncdisql101_bk.cernerasp.com 10.159.144.189 bpcd VIA pbx
        status: FAILED, (44) CONNECT_TIMEOUT; system: (115) Operation now in progress; FROM 170.71.1.193 TO cerncdisql101_bk.cernerasp.com 10.159.144.189 bpcd VIA vnetd
        status: FAILED, (44) CONNECT_TIMEOUT; system: (115) Operation now in progress; FROM 170.71.1.193 TO cerncdisql101_bk.cernerasp.com 10.159.144.189 bpcd
07:14:17.905 [15835] <8> vnet_connect_to_bpcd: [vnet_connect.c:279] connect_to_service() failed 18 0x12
07:14:17.905 [15835] <2> local_bpcr_connect: Can't connect to client cerncdisql101_bk.cernerasp.com
07:14:17.906 [15835] <2> ConnectToBPCD: bpcd_connect_and_verify(cerncdisql101_bk.cernerasp.com, cerncdisql101_bk.cernerasp.com) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(cerncdisql101_bk.cernerasp.com) failed: 25
07:14:17.906 [15835] <16> bptestbpcd main: Function ConnectToBPCD(cerncdisql101_bk.cernerasp.com) failed: 25
<2>bptestbpcd: cannot connect on socket
07:14:17.906 [15835] <2> bptestbpcd: cannot connect on socket
 

 

 

Naseer Shaik's picture

Hi,

 

I have posted my response 30 mins abck,it said like its sent for approval and wil paste it after approval.

How long will it take?

Marianne's picture

We actually needed to test reverse lookup here with -ip, not -hn:
bpclntcmd -hn 10.159.144.189
and here as well:
bpclntcmd -hn 10.159.128.17

It does seem as if name lookup is fine.

The fact that nothing is logged in bpcd log tells me that the firewall is blocking connection.
Is this Windows firewall or separate firewall?
If Windows Firewall, try to disable firewall completely.
If separate firewall, speak to firewall admins.

To test port connectivity, try to telnet to port 1556 from media server:

telnet cerncdisql101_bk.cernerasp.com 1556
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Naseer Shaik's picture

Lookup with Ip worked fine

C:\Program Files\VERITAS\NetBackup\bin>bpclntcmd -ip 10.159.128.17
host 10.159.128.17: taspmonbj02_bk.cernerasp.com at 10.159.128.17
aliases:     taspmonbj02_bk.cernerasp.com     10.159.128.17

root@taspmonbj02:/etc
07:16:34 # bpclntcmd -ip 10.159.144.189
host 10.159.144.189: cerncdisql101_bk.cernerasp.com at 10.159.144.189
aliases:     cerncdisql101_bk.cernerasp.com     10.159.144.189
root@taspmonbj02:/etc
02:25:09 # telnet cerncdisql101_bk.cernerasp.com 1556
Trying 10.159.144.189...
Connected to cerncdisql101_bk.cernerasp.com (10.159.144.189).
Escape character is '^]'.
 

need to check with admin for firewall issue

Naseer Shaik's picture

I dont know any commands or procedures to check firewall issues,by chance if you have any cmds/proc ,Please tell me the commands/procedure so that I can do primary checks without help of firewall Admin.

sorry,I know this is not right forum for firewall issues but just trying my luck if any one is having any handy doc.

Thank you in Advance.

Marianne's picture

If taspmonbj02 is your media server, you can see that telnet to port 1556 on client works fine. So, does not look like firewall issue.

If all the correct services are running on the client, there should definately be an entry in client's bpcd log after backup or bptestbpcd attempt (if bpcd folder exists under C:\Program Files\VERITAS\NetBackup\logs). 

Please try again and post log as File attachment.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Naseer Shaik's picture

I have attached bpcd log,below is job details,here taspmopd601 is storage server,taspmonbj02 is media server

 

2012 03:22:09 - Info nbjm (pid=28089) starting backup job (jobid=1172937) for client cerncdisql101_bk.cernerasp.com, policy mscs_cerncdisqlcl01_nodes, schedule Full
10/30/2012 03:22:09 - Info nbjm (pid=28089) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=1172937, request id:{E62F4A0C-226A-11E2-AEA1-54D7E561D3F9})
10/30/2012 03:22:09 - requesting resource kc_pda_stu_taspmopd601
10/30/2012 03:22:09 - requesting resource taspmonbj01.cernerasp.com.NBU_CLIENT.MAXJOBS.cerncdisql101_bk.cernerasp.com
10/30/2012 03:22:09 - requesting resource taspmonbj01.cernerasp.com.NBU_POLICY.MAXJOBS.mscs_cerncdisqlcl01_nodes
10/30/2012 03:22:09 - granted resource  taspmonbj01.cernerasp.com.NBU_CLIENT.MAXJOBS.cerncdisql101_bk.cernerasp.com
10/30/2012 03:22:09 - granted resource  taspmonbj01.cernerasp.com.NBU_POLICY.MAXJOBS.mscs_cerncdisqlcl01_nodes
10/30/2012 03:22:09 - granted resource  MediaID=@aaaad;DiskVolume=PureDiskVolume;DiskPool=kc_pda_dp_taspmopd601;Path=PureDiskVolume;StorageServer=taspmopd601.cernerasp.com;MediaServer=taspmonbj02.cernerasp.com
10/30/2012 03:22:09 - granted resource  kc_pda_stu_taspmopd601
10/30/2012 03:22:10 - estimated 0 kbytes needed
10/30/2012 03:22:10 - Info nbjm (pid=28089) started backup (backupid=cerncdisql101_bk.cernerasp.com_1351585329) job for client cerncdisql101_bk.cernerasp.com, policy mscs_cerncdisqlcl01_nodes, schedule Full on storage unit kc_pda_stu_taspmopd601
10/30/2012 03:22:10 - started process bpbrm (pid=17108)
10/30/2012 03:27:36 - Info bpbrm (pid=17108) connect failed STATUS (18) CONNECT_FAILED
10/30/2012 03:27:36 - Info bpbrm (pid=17108)  status: FAILED, (44) CONNECT_TIMEOUT; system: (115) Operation now in progress; FROM 170.71.1.193 TO cerncdisql101_bk.cernerasp.com 10.159.144.189 bpcd VIA pbx
10/30/2012 03:27:36 - Info bpbrm (pid=17108)  status: FAILED, (44) CONNECT_TIMEOUT; system: (115) Operation now in progress; FROM 170.71.1.193 TO cerncdisql101_bk.cernerasp.com 10.159.144.189 bpcd VIA vnetd
10/30/2012 03:27:36 - Info bpbrm (pid=17108)  status: FAILED, (44) CONNECT_TIMEOUT; system: (115) Operation now in progress; FROM 170.71.1.193 TO cerncdisql101_bk.cernerasp.com 10.159.144.189 bpcd
10/30/2012 03:27:36 - Error bpbrm (pid=17108) Cannot connect to cerncdisql101_bk.cernerasp.com
10/30/2012 03:27:36 - Info bpbkar (pid=0) done. status: 58: can't connect to client
10/30/2012 03:27:11 - end writing
can't connect to client  (58)

AttachmentSize
bpcdlog.txt 1.67 KB
CRZ's picture

All your troubles appear to involve attempts to get to hosts on the 10.159.x.x network but with your requests originating from 170.71.1.193.  I am guessing the 10.159 and 170.71 nets are not connected, by design.

Do you have an additional interface on your server which is on the backup network? 

Why are your servers trying to connect to your clients on the backup network while using its interface on the primary network?

These are the questions you need to figure out.