Video Screencast Help

Error 58. Can't Connect to Client

Created: 06 Apr 2013 • Updated: 13 Apr 2013 | 16 comments
This issue has been solved. See solution.

Hi All,

 

Master Server Sun OS 7.1.0.2

Media server Sun OS 7.1

Client : Unix 6.5.6

 

Backups are failing with error 58.

 

Bptestbpcd from Master is working fine..whereas it is failing from media server.

Below is the output of bptestbpcd from media server

08:15:33.076 [15167] <2> bptestbpcd: VERBOSE = 3
08:15:33.078 [15167] <2> ConnectionCache::connectAndCache: Acquiring new connection for host usprd144-ebr, query type 223
08:15:33.114 [15167] <2> vnet_pbxConnect: pbxConnectEx Succeeded
08:15:33.115 [15167] <2> logconnections: BPDBM CONNECT FROM 135.214.245.101.45769 TO 135.214.168.116.1556 fd = 4
08:15:33.132 [15167] <2> db_CLIENTsend: reset client protocol version from 0 to 7
08:15:33.252 [15167] <2> db_end: Need to collect reply
08:15:33.253 [15167] <2> db_freeEXDB_INFO: ?
08:18:45.280 [15167] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 145 0x00000091
08:18:57.282 [15167] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 145 0x00000091
08:19:09.283 [15167] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 145 0x00000091
08:20:34.290 [15167] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.978: 0: ran out of time before connect: 301 0x0000012d
08:20:34.291 [15167] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.978: 0: ran out of time before connect: 301 0x0000012d
08:20:34.291 [15167] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.978: 0: ran out of time before connect: 301 0x0000012d
08:20:34.291 [15167] <2> connect_to_service: ../../libvlibs/vnet_connect.c.382: 0: vnet_async_connect() failed: 18 0x00000012
08:20:34.291 [15167] <2> vnet_connect_to_bpcd: ../../libvlibs/vnet_connect.c.237: 0: connect_to_service() failed: 18 0x00000012
08:20:34.291 [15167] <2> local_bpcr_connect: Can't connect to client uhpctra2-ebr.madc.att.com
08:20:34.291 [15167] <2> ConnectToBPCD: bpcd_connect_and_verify(uhpctra2-ebr.madc.att.com, uhpctra2-ebr.madc.att.com) failed: 25
<16>bptestbpcd main: Function ConnectToBPCD(uhpctra2-ebr.madc.att.com) failed: 25
08:20:34.292 [15167] <16> bptestbpcd main: Function ConnectToBPCD(uhpctra2-ebr.madc.att.com) failed: 25
<2>bptestbpcd: cannot connect on socket
08:20:34.293 [15167] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
08:20:34.293 [15167] <2> bptestbpcd: EXIT status = 25
cannot connect on socket
 

Ping and telnet to media and client are working fine both the ways..

Please help

Operating Systems:

Comments 16 CommentsJump to latest comment

Marianne's picture

Maybe firewall between media server and client? If so, port 13724 (vnetd) needs to be open in both directions).

Have you verified that /etc/hosts entry exists on client for the media server?
And same hostname exist as SERVER entry in bp.conf (below SERVER entry for master server) on the client?

Please create bpcd folder on the client under /usr/openv/netbackup/logs, retry bptestbpcd from media server. Post bpcd log file.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

nbu123's picture

vnetd ports are open..

Bp.conf has entry of the media server.

Please find the bpcd logs

AttachmentSize
bpcd.txt 60.8 KB
Dan@NB's picture

> Check proper routes exists at master and media servers for client server.

> Ensure ports 13782, 13724 and 1556 are open and have bi-directional connectivity from master, media and client servers. If not, have your fierwall team address the issue.

> Ensure bp.conf on client server has entry of master and media servers.

> Ensure entry of master and media server in /etc/hosts file for client server and vice versa in the following format:

<IP Address>            <FQDN of the server>             <Short name of the server>

> Ensure correct hostname entry (as in client side) has been made in Master server host properties.

Above points should resolve the issue.

 

+ Dan

 

Marianne's picture

 

02:34:11.325 [25243] <2> logconnections: BPCD ACCEPT FROM 135.214.233.131.50543 TO 135.101.224.179.13782
02:34:11.326 [25243] <2> bpcd main: setup_sockopts complete
02:34:11.329 [25243] <2> Connection from host usprd144.edc.cingular.net (135.214.233.131) 

 

02:34:11.336 [25243] <4> bpcd valid_server: usprd144.edc.cingular.net is not a master server
02:34:11.336 [25243] <16> bpcd valid_server: usprd144.edc.cingular.net is not a media server either
02:34:12.846 [25243] <16> bpcd main: read failed: Can't assign requested address

 

Please add SERVER entry in client's bp.conf for usprd144.edc.cingular.net.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

nbu123's picture

As suggested i have added the entry to the client bp.conf...Still its failing with same error..

 

Marianne's picture

Please rename current bpcd log on client to ensure that we see only one connection attempt in the new log that will be created.

Run bptestbpcd from the media server with -verbose -debug options:
bptestbpcd -client <client-name> -verbose -debug

Post output of command as well as client's bpcd log.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

nbu123's picture

Hi Marianne,

 

Please find the output of bpcd log from client and output of bptestbpcd from media to client

AttachmentSize
bpcd.txt 14.04 KB
bptestbpcd.txt 3.57 KB
captain jack sparrow's picture

Try refreshing hostcache and recycling NBU services + PBX service if it helps

 

 Cheers !!!

CJS

 

nbu123's picture

Cleared the Hostcache still no luck...

 

Also recyling NBU will not help as other clients using the Media server are working properly..

Marianne's picture

Let us start from the beginning please?

What is the hostname and IP address of problematic media server?

It seems you did not use -verbose and -debug with bptestbpcd and did not rename client's bpcd log before testing connection. 

We see only successful connection attempt from media server usprd144-nebr.madc.att.com  in client's bpcd log:

 

Connection from host usprd144-nebr.madc.att.com (10.112.184.81)

Because bptestbpcd output does not contain enough information, we cannot see which IP address the media server is trying to connect to or which port it is trying to connect to.

Please show us output of ALL of the following on the media server:

bpclntcmd -self

bpclntcmd -hn uhpctra2-ebr.madc.att.com

ping uhpctra2-ebr.madc.att.com

traceroute uhpctra2-ebr.madc.att.com

telnet uhpctra2-ebr.madc.att.com 13724

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

nbu123's picture

Media server in question is usprd134.madc.att.com

 

We are tried to ping the client's backup interface and telnet the BPCD port from Media server its not working.

Hence i am checking with network team to correct this problem....Will update once communication from Media to client is working...

nbu123's picture

Hi Marianne,

Finally Connectivity has been established between Media and Client.Still backups are failing.

Please find the below details

Media server Name: usprd134,

Master Name: usprd144

Client Name:uhpctra2-ebr.madc.att.com

Output from Media Servers:-

bpclntcmd -self
yp_get_default_domain failed: (6) internal yp server or client error
NIS does not seem to be running: (1) args to yp function are bad
gethostname() returned: usprd134-ebr
host usprd134-ebr: usprd134-ebr.edc.cingular.net at 135.214.245.101
host usprd134-ebr: usprd134-ebr.edc.cingular.net at 135.214.245.103
host usprd134-ebr: usprd134-ebr.edc.cingular.net at 135.214.245.102
host usprd134-ebr: usprd134-ebr.edc.cingular.net at 135.214.245.107
host usprd134-ebr: usprd134-ebr.edc.cingular.net at 135.214.245.108
host usprd134-ebr: usprd134-ebr.edc.cingular.net at 135.214.245.105
host usprd134-ebr: usprd134-ebr.edc.cingular.net at 135.214.245.106
host usprd134-ebr: usprd134-ebr.edc.cingular.net at 135.214.245.104
aliases:     usprd134-ebr.edc.cingular.net     usprd134-ebr     135.214.245.108     135.214.245.107     135.214.245.106     135.214.245.105     135.214.245.104     135.214.245.103     135.214.245.102     135.214.245.101
 

bpclntcmd -hn uhpctra2-ebr.madc.att.com
host uhpctra2-ebr.madc.att.com: uhpctra2-ebr.madc.att.com at 10.112.184.52
aliases:     uhpctra2-ebr.madc.att.com     10.112.184.52

 ping uhpctra2-ebr.madc.att.com
uhpctra2-ebr.madc.att.com is alive
 

traceroute uhpctra2-ebr.madc.att.com
traceroute: Warning: Multiple interfaces found; using 10.112.0.11 @ nxge1
traceroute to uhpctra2-ebr.madc.att.com (10.112.184.52), 30 hops max, 40 byte packets
 1  allntxdwgradcnx01-v400.dc.swst.att.com (10.112.0.1)  0.732 ms  0.517 ms  0.351 ms
 2  uhpctra2-ebr.madc.att.com (10.112.184.52)  0.368 ms  0.343 ms  0.399 ms
 

telnet uhpctra2-ebr.madc.att.com 13724
Trying 10.112.184.52...
Connected to uhpctra2-ebr.madc.att.com.

bptestbpcd -verbose -debug -client uhpctra2-ebr
04:23:11.331 [29288] <2> bptestbpcd: VERBOSE = 3
04:23:11.332 [29288] <2> ConnectionCache::connectAndCache: Acquiring new connection for host usprd144-ebr, query type 223
04:23:11.351 [29288] <2> vnet_pbxConnect: pbxConnectEx Succeeded
04:23:11.351 [29288] <2> logconnections: BPDBM CONNECT FROM 135.214.245.101.63077 TO 135.214.168.116.1556 fd = 4
04:23:11.369 [29288] <2> db_CLIENTsend: reset client protocol version from 0 to 7
04:23:11.490 [29288] <2> db_getCLIENT: db_CLIENTreceive: no entity was found 227
04:23:11.492 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
04:23:11.492 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: /usr/openv/var/host_cache/161/1cea2f61+0,1,50,0,2,0+uhpctra2-ebr.txt
04:23:11.497 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
04:23:11.497 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: /usr/openv/var/host_cache/161/1cea2f61+veritas_pbx,1,4,2,2,0+uhpctra2-ebr.txt
04:23:11.500 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
04:23:11.500 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: /usr/openv/var/host_cache/167/e3c00f67+0,1,50,0,2,0+10.112.184.52.txt
04:23:11.504 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
04:23:11.504 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: /usr/openv/var/host_cache/161/1cea2f61+vnetd,1,4,2,2,0+uhpctra2-ebr.txt
04:23:11.507 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6635: 0: fopen() failed: 2 0x00000002
04:23:11.507 [29288] <2> file_to_addrinfo: ../../libvlibs/vnet_addrinfo.c.6636: 0: fopen() failed: /usr/openv/var/host_cache/161/1cea2f61+bpcd,1,4,2,2,0+uhpctra2-ebr.txt
04:23:11.519 [29288] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1376: 0: getsockopt SO_ERROR returned: 146 0x00000092
04:23:11.588 [29288] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 150 0x00000096
04:23:11.588 [29288] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function failed: 9 0x00000009
04:23:11.588 [29288] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Function failed: 9 0x00000009
04:23:11.588 [29288] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Functio
04:23:11.588 [29288] <2> vnet_version_connect: ../../libvlibs/vnet_vnetd.c.1659:
04:23:11.588 [29288] <2> do_vnetd_service: ../../libvlibs/vnet_connect.c.1610: 0
04:23:11.588 [29288] <2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1353:
04:23:11.599 [29288] <2> logconnections: BPCD CONNECT FROM 10.112.0.11.738 TO 10
04:23:11.601 [29288] <2> vnet_vnetd_push_ipaddr: ../../libvlibs/vnet_vnetd.c.178
04:23:11.601 [29288] <2> vnet_vnetd_push_ipaddr: ../../libvlibs/vnet_vnetd.c.178
04:23:11.601 [29288] <2> vnet_vnetd_push_ipaddr: ../../libvlibs/vnet_vnetd.c.178
04:23:11.601 [29288] <2> vnet_vnetd_push_ipaddr: ../../libvlibs/vnet_vnetd.c.178
04:23:12.170 [29288] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1159: 0: errno: 2
04:23:12.170 [29288] <2> vnet_pop_byte: ../../libvlibs/vnet.c.1161: 0: Function
04:23:12.170 [29288] <2> vnet_pop_string: ../../libvlibs/vnet.c.1241: 0: Functio
04:23:12.170 [29288] <2> vnet_pop_signed: ../../libvlibs/vnet.c.1285: 0: Functio
04:23:12.170 [29288] <2> vnet_pop_status: ../../libvlibs/vnet.c.1363: 0: Functio
04:23:12.170 [29288] <2> vnet_begin_connect_back: ../../libvlibs/vnet_vnetd.c.61
04:23:12.170 [29288] <2> ConnectToBPCD: bpcd_connect_and_verify(uhpctra2-ebr, uh
<16>bptestbpcd main: Function ConnectToBPCD(uhpctra2-ebr) failed: 25
04:23:12.171 [29288] <16> bptestbpcd main: Function ConnectToBPCD(uhpctra2-ebr)
<2>bptestbpcd: cannot connect on socket
04:23:12.171 [29288] <2> bptestbpcd: cannot connect on socket
<2>bptestbpcd: EXIT status = 25
04:23:12.171 [29288] <2> bptestbpcd: EXIT status = 25
cannot connect on socket
 

Output from Clients BPCD Log

cat log.041313
01:39:55.924 [18013] <16> bpcd main: ParseArguments failed
01:44:56.977 [27764] <16> bpcd main: ParseArguments failed
04:23:12.163 [21847] <16> bpcd main: ParseArguments failed

 

Please let me in case any other information is required
 

Ankit Maheshwari's picture

 

Step by step process need to follow...check ping status...

 

1-Do one thing.. Upgrade client to 7.1...

2-/etc/services system configuration file can be used to

override the default port numbers for each port so please check entry for bpcd, vnetd and pbx port.

 

3-Check bp.conf, /etc/hosts.

4-Perform below task from Master Media and client.

 

bpclntcmd -hn hostname -verbose

bpclntcmd -ip ipadress -verbose.

 

5-Check weather port is open or not...

telnet hostname portnumber

 

 

Ankit

 

 

Ankit Maheshwari

Marianne's picture

Ping should always be first step to test network comms.
As long as that is not working, forget about NBU.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Dan@NB's picture

nbu123,

Please ensure PBX is running for the client server. Most of the time it creates several issues. Also confirm is bi-directional connectivity exists between master, media and client server.

Also multiple entry of client server in /etc/hosts of master and media can lead to this issue. Single entry in /etc/hosts with correct IP address (confirm correct IP with server team if they changed it for some activity on the server) , FQDN and short name.

Check and let us know.

 

+ Dan

nbu123's picture

I figure it by own..

Restarted the inted services and now backups are working fine...

Thanks to all for your kind help.. :-)

SOLUTION