Video Screencast Help

XBSA client <-> server backup failure

Created: 06 Feb 2013 • Updated: 06 Feb 2013 | 7 comments

Hi,

Since we upgraded to Netbackup v7.1.0.3, the Netbackup client fails to communicate to the server via XBSA (AIX platform). Or more precisely, the backup fails, since the communication seems first successful (C_GET_FEATURES command OK) but fails after that. There is no firewall involved. Strangely, all backup demands via XBSA are now frozen (the last log in exten_client is "connect in progress") and no answer is ever received. However, a standard client backup works. When the Netbackup client is in v6.x, it works also !

End of log of bprd (server side) :

16:44:37.683 [13642] <2> vnet_pbxAcceptSocket: Accepted sock[9] from IP9:34428
16:44:37.683 [13642] <2> bprd: socket fd from accept() is 9
16:44:37.684 [13642] <2> listen_loop: request complete
16:44:37.685 [17454] <2> logconnections: BPRD ACCEPT FROM IP9.34428 TO IP129.1556 fd = 9
16:44:37.686 [17454] <2> process_request: setsockopt SO_LINGER on 9 succeeded.
16:44:37.686 [17454] <2> connected_peer: Connection from host CLIENTNAME_HIDDEN, IP9, on non-reserved port 34428
16:44:37.688 [17454] <2> process_request: command C_GET_FEATURES (72) received
16:44:37.691 [17454] <2> process_request: EXIT STATUS 0
16:44:37.697 [13642] <2> vnet_pbxAcceptSocket: Accepted sock[9] from IP9:34430
16:44:37.697 [13642] <2> bprd: socket fd from accept() is 9
16:44:37.697 [13642] <2> listen_loop: request complete
16:44:37.698 [17455] <2> logconnections: BPRD ACCEPT FROM IP9.34430 TO IP129.1556 fd = 9
16:44:37.699 [17455] <2> process_request: setsockopt SO_LINGER on 9 succeeded.
16:44:37.699 [17455] <2> connected_peer: Connection from host CLIENTNAME_HIDDEN, IP9, on non-reserved port 34430
16:44:37.811 [17455] <2> get_long: (2) premature end of file (byte 1)
16:44:37.816 [17455] <32> process_request: get_string() failed - premature end of file encountered (5)
16:44:37.820 [17455] <32> process_request: could not process request from CLIENTNAME_HIDDEN
16:44:38.829 [13642] <2> ConnectionCache::connectAndCache: Acquiring new connection for host SERVERNAME_HIDDEN, query type 98
16:44:38.829 [13642] <2> logconnections: BPDBM CONNECT FROM IP68.46871 TO IP68.13721 fd = 9
16:44:38.829 [13642] <2> db_end: Need to collect reply

End of log of exten_client (client side):

<2> vnet_cached_getaddrinfo_and_update: ../../libvlibs/vnet_addrinfo.c.1373: 0: found in cache service: NULL
<2> get_pref_netconnection: ../../libvlibs/vnet_addrinfo.c.5032: 0: prefnet = : REQIF
<2> vnet_async_connect: ../../libvlibs/vnet_connect.c.1200: 0: connect in progress: 1 0x00000001

The bpclntcmd command on the client seems successful :

 bpclntcmd -hn SERVERNAME_HIDDEN
host SERVERNAME_HIDDEN: SERVERNAME_HIDDEN at IP129
aliases:     SERVERNAME_HIDDEN     IP129

bpclntcmd -hn CLIENTNAME_HIDDEN
host CLIENTNAME_HIDDEN: CLIENTNAME_HIDDEN at IP9
aliases:     CLIENTNAME_HIDDEN     IP9

 bpclntcmd -self
yp_get_default_domain failed: (6) internal yp server or client error
NIS does not seem to be running: (1) args to yp function are bad
gethostname() returned: CLIENTNAME_HIDDEN
host CLIENTNAME_HIDDEN: CLIENTNAME_HIDDEN at IP9
aliases:     CLIENTNAME_HIDDEN     IP9

bpclntcmd -ip IP129
host IP129: SERVERNAME_HIDDEN at IP129
aliases:     SERVERNAME_HIDDEN     IP129

Any idea ? I have attached logs files.

Thanks for your help.

Comments 7 CommentsJump to latest comment

Mark_Solutions's picture

It looks like this may be a Symantec Private Branch Exchange (pbx or pbx_exchange) issue

As you see in your log file the connection goes over port 1556 which is the pbx port

It may be that your client is having an issue with running pbx_exchange on it when at V7.1

Check that it is running, it is located at /opt/VRTSpbx/bin/

You can check its status using this doc http://www.symantec.com/docs/TECH44074

To check its logs use:

vxlogview -o 103 -t 02:00:00

This will pipe out the pbx logs for the last 2 hours so if it is not running start it and then take a look at its logs

7.1 also does a lot more with reverse lookups so hosts files can be useful to make sure the client can communicate correctly with the Master and Media Servers and vice versa, using short and FQDN

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

mikael_mechoulam's picture

I realize I didn't properly "reply" to your answer, but answered in another comment. In case you were hence not notified of my answer, please find it later in this thread.

Thanks.

mikael_mechoulam's picture

Thanks, we are going to check that, but why, if it is really a problem of pbx, is the following log present in extenclient :

vnet_pbxConnect: pbxConnectEx Succeeded

and why "normal" client backups (not using XBSA) work without pbx working correctly ?

 

mikael_mechoulam's picture

Hi,

We have performed the checks on pbx. The pbx process is running normally on the master and on the client, the /etc/services contains the service on port 1556. The pbxcfg seems to be normal, both on server and client.

/opt/VRTSpbx/bin/pbxcfg -p (client)
Auth User:0 : root
Secure Mode: false
Debug Level: 10
Port Number: 1556
PBX service is not cluster configured

However, when archiving through XBSA, there is no log on client side generated by pbx. The process seems inactive :

/opt/VRTSpbx/bin # ps -ef |grep pbx 
  root  491588       1   0   Jan 14      -  0:00 /opt/VRTSpbx/bin/pbx_exchange 

What should we do now ?

Thanks,

mikael_mechoulam's picture

Hi,

Some last details, if it can help. We have shutdown the pbx service on client side, it changes nothing. The command bptestnetconn on the client returns right results. A telnet to 1556 port from the client to the server works also. It seems really, from the log of exten_client, that PBX hangs : a first connection is done

vnet_async_connect: ../../libvlibs/vnet_connect.c.1200: 0: connect in progress: 1 0x00000001
vnet_pbxConnect: pbxConnectEx Succeeded

but a second one is blocked : 

vnet_async_connect: ../../libvlibs/vnet_connect.c.1200: 0: connect in progress: 1 0x00000001

and nothing after that.

We see the traces of two connections in the PBX logs server side :

03/01/2013 09:37:08.294 [Info] PBX_Manager:: handle_input with fd = 7
03/01/2013 09:37:08.294 [Info] PBX_Client_Proxy::parse_line, line = ack=1  From IP9
03/01/2013 09:37:08.294 [Info] PBX_Client_Proxy::parse_line, line = extension=bprd  From IP9
03/01/2013 09:37:08.294 [Info] hand_off looking for proxy for = bprd
03/01/2013 09:37:08.294 [Info] is_accepting about to return true
03/01/2013 09:37:08.294 [Info] Client Expects an ACK, sent one.
03/01/2013 09:37:08.294 [Info] Proxy found.
03/01/2013 09:37:08.294 [Info] PBX_Client_Proxy::handle_close
03/01/2013 09:37:08.305 [Info] PBX_Manager:: handle_input with fd = 7
03/01/2013 09:37:08.306 [Info] PBX_Client_Proxy::parse_line, line = ack=1  From IP9
03/01/2013 09:37:08.306 [Info] PBX_Client_Proxy::parse_line, line = extension=bprd  From IP9
03/01/2013 09:37:08.306 [Info] hand_off looking for proxy for = bprd
03/01/2013 09:37:08.306 [Info] is_accepting about to return true
03/01/2013 09:37:08.306 [Info] Client Expects an ACK, sent one.
03/01/2013 09:37:08.306 [Info] Proxy found.
03/01/2013 09:37:08.306 [Info] PBX_Client_Proxy::handle_close

I've seen http://clientui-kb.symantec.com/kb/index?page=content&pmv=print&impressions=&viewlocale=&id=TECH174379 but it doesn't seem to be related (there are not so many connections).

Still no idea ?

Thanks,

Marianne's picture

Check that this command on client returns expected client name from master:

bpclntcmd -pn

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mikael_mechoulam's picture

Hi,

Yes, this seems the case :

SERVERNAME_HIDDEN@root:/opt/VRTSpbx/bin # bpclntcmd -pn
expecting response from server CLIENTNAME_HIDDEN
CLIENTNAME_HIDDEN CLIENTNAME_HIDDEN IP9 46025 

What should we check now ? Do you know PBX-related fixes from 7.0.3 to 7.5.x that could be linked to this problem, apart from 2525096 ?

Thanks for your help,