Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

Exit Status 58 - New message what I've never seen

Created: 08 Apr 2013 | 7 comments

Offlate I'm seeing many clients failing with exit statys '58' though configurations looks good for me. Primary error message from the master server is

 

status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx

        status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd

status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx

        status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd

 

Verified the logs and it mostly looks for me a client level firewall issue but I disabled all the firewall with no luck and same error. Any help is much appreciated.

Here are the test bpcd logs from master and corresponding logs from the client.

 

 

TEST-BPCD from master results:

 

D:\Veritas\NetBackup\bin\admincmd>bptestbpcd -client lcspepas56rnb -verbose -deb

ug

11:30:28.281 [4868.9624] <2> bptestbpcd: VERBOSE = 0

11:30:28.294 [4868.9624] <2> vnet_pbxConnect: pbxConnectEx Succeeded

11:30:28.296 [4868.9624] <2> logconnections: BPCD CONNECT FROM 10.186.240.20.586

37 TO 10.186.242.56.1556 fd = 608

11:30:28.309 [4868.9624] <2> vnet_pbxConnect: pbxConnectEx Succeeded

11:35:28.323 [4868.9624] <8> vnet_sock_ready: [vnet.c:1485] max_time=300 sock=63

11:35:28.340 [4868.9624] <16> connect_to_service: connect failed STATUS (18) CON

NECT_FAILED

        status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA pbx

        status: FAILED, (11) TIMEOUT; system: (10036) A blocking operation is cu

rrently executing. ; FROM 0.0.0.0 TO 10.186.242.56 10.186.242.56 vnetd VIA vnetd

 

11:35:28.341 [4868.9624] <8> vnet_connect_to_vnetd_ipi: [vnet_connect.c:258] con

nect_to_service() failed 18 0x12

11:35:28.341 [4868.9624] <16> bpcr_vnetd_connect_forward_socket_begin: vnet_conn

ect_to_vnetd_ipi(10.186.242.56) failed: 18

11:35:28.341 [4868.9624] <2> local_bpcr_connect: bpcr_vnetd_connect_forward_sock

et_begin failed: 25

11:35:28.342 [4868.9624] <2> ConnectToBPCD: bpcd_connect_and_verify(lcspepas56rn

b, lcspepas56rnb) failed: 25

<16>bptestbpcd main: Function ConnectToBPCD(lcspepas56rnb) failed: 25

11:35:28.343 [4868.9624] <16> bptestbpcd main: Function ConnectToBPCD(lcspepas56

rnb) failed: 25

<2>bptestbpcd: cannot connect on socket

11:35:28.343 [4868.9624] <2> bptestbpcd: cannot connect on socket

<2>bptestbpcd: EXIT status = 25

11:35:28.343 [4868.9624] <2> bptestbpcd: EXIT status = 25

cannot connect on socket

 

 

Client BPCD logs for the above test:

 

14:10:12.282 [2452.7104] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeBackupPrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeDebugPrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeRestorePrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeSecurityPrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeTakeOwnershipPrivilege)

14:10:12.282 [2452.7104] <2> CrankUpNTPrivs: About to EnablePrivilege(SeTcbPrivilege)

14:10:12.282 [2452.7104] <2> setup_debug_log: switching debug log file for bpcd

14:10:12.282 [2452.7104] <2> setup_debug_log: switched debug log file for bpcd

14:10:12.298 [2452.7104] <2> daemon_startup_listeners: 4 listening for legacy service bpcd

14:10:12.298 [2452.7104] <2> daemon_startup_listeners: listening for vnetd service bpcd

14:10:12.313 [2452.7104] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.132: pbxRegisterEx successful at 10.186.242.56:5126/bpcd, returns with 1 alt_addrs

14:10:12.313 [2452.7104] <2> vnet_registerPBXServer: ../../libvlibs/vnet_pbx.c.144: alt_addr: 10.186.163.91:5126

14:15:31.581 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

14:30:34.868 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

14:32:59.291 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

14:35:35.057 [2452.7104] <2> vnet_pbxAcceptSocket: invalid socket passed to getpeername 10022:An invalid argument was supplied.  )

 

Client VNETD logs for above test:

 

12:58:33.645 [984.852] <2> main: VERBOSE = 0

12:58:33.645 [984.852] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\vnetd.exe -standalone

12:58:33.661 [984.852] <8> ProcessRequests: [.\vnetd.c:635] msg VNETD ACCEPT FROM 10.186.240.20.59478 TO 10.186.242.56.13724 fd = 520

12:58:33.661 [984.852] <8> vnet_pop_byte: [vnet.c:1168] errno 0 0x0

12:58:33.661 [984.852] <2> vnet_pop_byte: vnet.c.1170: 0: Function failed: 9 0x00000009

12:58:33.661 [984.852] <2> vnet_pop_string: vnet.c.1250: 0: Function failed: 9 0x00000009

12:58:33.661 [984.852] <2> vnet_pop_signed: vnet.c.1293: 0: Function failed: 9 0x00000009

12:58:33.661 [984.852] <2> vnet_version_accept: .\vnetd.c.1089: 0: Function failed: 9 0x00000009

12:58:33.661 [984.852] <8> ProcessRequests: [.\vnetd.c:640] version_accept failed 9 0x9

12:58:33.661 [984.852] <8> main: [.\vnetd.c:405] ProcessRequests failed 9 0x9

12:58:33.661 [984.852] <8> main: [.\vnetd.c:406] ProcessRequests returned 9 0x9

13:06:33.804 [5624.684] <2> setup_debug_log: switched debug log file for vnetd

13:06:33.804 [5624.684] <2> main: VERBOSE = 0

 

 

 

 

 

Operating Systems:

Comments 7 CommentsJump to latest comment

RamNagalla's picture

Does your windows filrewall is turned off.. 

its more likely fire wall issue?

Marianne's picture

Couple of questions:

 

Has this worked previously?
Or is this a new installation that has never worked before?

Have you confirmed that Windows Firewall is disabled on this machine?

Have you checked for third-party software that could be blocking or preventing port connection and/or socket setup?
Some Anti-Virus software is doing this. We have even seen that Download Manager sofware was preventing socket setup.
Some other ideas as to what third-part software to check for:
http://www.symantec.com/docs/TECH88628

We don't see TCP/IP 10038 error in your bpcd log, but worth a look.

We do see TCP/IP 10022 error in your log. I have Googled the error which points to misconfigured sofware or even 'improperly-formatted IPv4 or IPv6 Network Address'.

Perhaps multiple NICs on this client?

If so, does each NIC/IP address have a different hostname?

Have you tried to remove and reinstall NBU software?

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Nithin S's picture

Nagalla - Windows firewall was already disabled and also disabled the IP Sec service and rebooted the machine with no luck.

Marianne - This was all working since 6-7 days back. Windows Firewall and IP sec disabled. I also tried disabling McAfee for few moment. As you see, I can telnet to all ports 1556, 13724, 13782 and also ping the client from the master. Client also able to resolve the master/media back.

I can again check on the McAfee part.

There are 10038 but that's for a different client (I've many failing:-)). For this client only 10022 in the logs.

We do have couple of NICs and one for production and other dedicated for backup and each NIC having a different DNS name. We do DNS resolution and resolution seems to be good. I can uninstall and try to reinstall again, yes.

 

 

Mark_Solutions's picture

If you have renamed the McAfee forewall driver you will need to reboot the client for that to take effect

Also check the McAfee Access Protection as that can also block it.

See this tech note for everything you need to do:

http://www.symantec.com/docs/TECH56658

I have seen some recent updates seem to re-set the McAfee settings so they need to be re-applied

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Nithin S's picture

I tried to play around with McAfee but no luck. If McAfee is blocking, it should appear in McAfee access protection logs and I don't see anything related to NBU there.

Last time when McAfee created trouble, telnet from master to client over 1556/13782 or 13724 never worked but presently, that's all working without any issues. I've opened an SYMC case anyway as more client appear to have same issue and it's happening only for Windows 2003 machines.

 

 

Nithin S's picture

After working with Symantec support,    Under Master server properties ==> Firewall

Added the host and set as below

BPCD Connect back = VNET port
Ports = Reserved port
Daemon connection port = Daemon port only

Looks like "Daemon port only" did the trick but I'm still not sure why it's not working with VNETD port and we have the 1556 port opened bidirectional

Windows 2008 client on same subnet works fine without above settings but only Windows 2003 is having issues. Not sure if Windows 2003 is having a compatibility issue with BPX, does any of you know?

Marianne's picture

Which NBU version (and patch level) on W2003 clients?

Check if Clients can telnet to vnetd and/or PBX ports on the master and media servers.

Please share W2003 client's bpcd log as File attachment.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links