Video Screencast Help

Windows Server 2012 & 7.5.0.6

Created: 05 Sep 2013 • Updated: 10 Sep 2013 | 28 comments
This issue has been solved. See solution.

Hi,

I have installed the Win2012 7.5.0.6 client onto a Win Server 2012 client but am having difficulties connecting from the Master Server to the client.

The install completed wihout error and I can run the Backup/Restore client from the Server; however, when I try to connect to the client from the Master GUI I get Connection Requested By Invalid Server, the Master was correctly placed in the relevant box on install and I have double checked the client GUI and this is correct.

If I run the usual connection tests from the master server I get the following:

bpclntcmd -hn lonbfbvmconn2

host lonbfbvmconn2: LONBFBVMCONN2.corp.ad.timeinc.com at 10.132.113.127

aliases:     LONBFBVMCONN2.corp.ad.timeinc.com     lonbfbvmconn2     10.132.113.127

telnet lonbfbvmconn2 bpcd

Trying 10.132.113.127...

Connected to lonbfbvmconn2.corp.ad.timeinc.com (10.132.113.127).

Escape character is '^]'.

bptestbpcd -host lonbfbvmconn2   

<16>bptestbpcd main: Function ConnectToBPCD(lonbfbvmconn2) failed: 46

server not allowed access

bfbackup.ipcmedia.com:/root> bptestbpcd -client lonbfbvmconn2

<16>bptestbpcd main: Function ConnectToBPCD(lonbfbvmconn2) failed: 46

server not allowed access

So as you can see I can quiet clearly access the bpcd port via Telnet and the bpclntcmd returns the correct client details but the bptstbpcd fails.There is no firewall active on the client and the client and master can do a reverse lookup on each other and can ping also.

Anyione got any ideas as I am at a loss

Kev

Operating Systems:

Comments 28 CommentsJump to latest comment

Mark_Solutions's picture

Setup a bpcd log on the client to see why it actually denies access

It may just be a FQDN thing and you will then need to either add a hosts file entry to the client and / or Master / Media Servers or add the FQDN to the clients server list

Create the bpcd directory - try and connect to it again and post the resulting log file

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Kevin Lamb's picture

Just done that, looking through the log (attached) it is stating that my master server is not a master or a media.....

AttachmentSize
090513.txt 12.16 KB

Kevin Lamb's picture

Just had a look at the Reg Key in HKEY_LOCAL_MACHINE\Softeare\veritas\netbackup\currentversion\config and the Server key includes the following:

bfbackup.ipcmedia.com
bfbackup
10.132.112.22

So it is listed in the software as a server in all the FQDN ways

chack22's picture

What does bptestbpcd show? I don't recall the options off the top of my head, but it checks connections to clients. What about bpclntcmd -hn (option) and bpclntcmd -ip (IP)? This checks lookups (forward and reverse) from the NBU side.

Kevin Lamb's picture

Chack22 if you take a look at the initial post the bpclntcmd returns ok but the bptesbpcd fails with server not allowed access, the telnet to the bpcd port works fine, all entries on the client are correct and I have even put the master in the host file and can do forward & reverse DNS from client to master and master to client.

CRZ's picture

I may be nuts, but something stood out from your log...

17:22:30.901 [2060.2032] <2> bpcd peer_hostname: Connection from host bfbackup.ipcmedia.com  (10.132.112.22) port 51386
...
17:22:31.151 [1108.536] <4> bpcd valid_server: bfbackup.ipcmedia.com  is not a master server
17:22:31.151 [1108.536] <16> bpcd valid_server: bfbackup.ipcmedia.com  is not a media server either

It sure looks like there's an invisible space appended to the reverse DNS of your master (I'm seeing two spaces instead of the one which I would expect) which is causing NetBackup to fail to match the name "bfbackup.ipcmedia.com " to "bfbackup.ipcmedia.com" because, to NetBackup, the strings ARE technically different...

Can you recheck the host file on the client and see if there's something you can edit?  (I'm assuming this isn't a problem for other clients so it's something local to this particular one, but you probably know your DNS better than I do)


bit.ly/770LBN | APPLBN | 761LBN

Kevin Lamb's picture

Hi,

I have checked and rewritten the hosts file on the Win2012 server and this still fails, I even removed the hosts file and checked I could still do a reverse nslookup to the Master from the client and thsi was Ok.

Still getting the same entries in the bpcd log about the master and media server.

Kev

Mark_Solutions's picture

As your entries look OK try this:

1. See what the Browser registry key is set to - this should match the Client_Name key

2. Run this twice on the client:

\program files\veritas\netbackup\bin\blclntcmd -clear_host_cache

3. Check what the Port_BPCD key value is - it should be 13782

4. Re-start the client service on the client

5. Try to connect again

Also on the client run bpclntcmd -sv to see what it brings back

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Kevin Lamb's picture

Hi Mark

The Reg keys look correct (see attached doc) I have ran the clear_cache option twice and restarted the client services, also the output of the bpclntcmd -sv returns 7.5

I am at a loss as to why it cannot relate to the Master Server

Kev

AttachmentSize
bpcd_test.docx 207.33 KB

Mark_Solutions's picture

There doesn't look to be enough components in the registry there

Could you try re-installing (run a repair) on the client - make sure AV is turned off and that you do everything using the Run As Administrator

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Kevin Lamb's picture

Uninstalled the client then ran a clean install as Admin, same issue

I have just removed the client and then cleared all relevant directories and cleaned the registry, rebooting the server, going to run another clean install and see what happens, not confident though

Mark_Solutions's picture

OK - do you have a Media Server too?

If so add that to its configuration and see if you can connect from that

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Kevin Lamb's picture

I have two media server, one Windows and one Linux, I did add both to the config and they both returned the same fault as the master where the server is refused connection.

Mark_Solutions's picture

OK - make sure you have the bpcd and bpinetd log folders - increase logging (via its BAR GUI) - re-start the services and try to connect again

Then post the logs

And just in case .. try setting the NetBackup Client Service and NetBackup Legacy Network Service to use a Domain Admin account to see if that makes any difference

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Kevin Lamb's picture

Set logging to 2, used a domain admin account to start the services, still get the same issue though :-(

Log files are attched

AttachmentSize
bpcd_log.txt 34.5 KB
bpinetd_log.txt 8.42 KB

Kevin Lamb's picture

Just ran the  bptestbpcd -client lonbfbvmconn2 -debug from my linux media server (lonbfbreddev) and this returns with a clean connection:

12:36:24.104 [25420] <2> bptestbpcd: VERBOSE = 0
12:36:24.104 [25420] <2> ConnectionCache::connectAndCache: Acquiring new connect
ion for host bfbackup.ipcmedia.com, query type 223
12:36:24.107 [25420] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:36:24.107 [25420] <2> logconnections: BPDBM CONNECT FROM 10.132.126.144.60212
 TO 10.132.112.22.1556 fd = 3
12:36:24.107 [25420] <8> vnet_check_vxss_client_magic_with_info: [vnet_vxss_help
er.c:871] Ignoring VxSS authentication 2 0x2
12:36:24.112 [25420] <2> db_CLIENTsend: reset client protocol version from 0 to 
8
12:36:24.153 [25420] <2> db_getCLIENT: db_CLIENTreceive: no entity was found 227
12:36:24.159 [25420] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:36:24.159 [25420] <2> logconnections: BPCD CONNECT FROM 10.132.126.144.52845 
TO 10.132.113.127.1556 fd = 3
12:36:24.169 [25420] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:36:24.219 [25420] <8> do_pbx_service: [vnet_connect.c:2108] via PBX VNETD CON
NECT FROM 10.132.126.144.49482 TO 10.132.113.127.1556 fd = 4
12:36:24.219 [25420] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:
443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
12:36:24.284 [25420] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:
460] ipc_string 64422
1 1 1
10.132.126.144:52845 -> 10.132.113.127:1556
10.132.126.144:49482 -> 10.132.113.127:1556
12:36:28.976 [25420] <2> vnet_pbxConnect: pbxConnectEx Succeeded
12:36:29.036 [25420] <8> do_pbx_service: [vnet_connect.c:2108] via PBX VNETD CON
NECT FROM 10.132.126.144.44483 TO 10.132.113.127.1556 fd = 5
12:36:29.036 [25420] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:
443] VN_REQUEST_CONNECT_FORWARD_SOCKET 10 0xa
12:36:29.104 [25420] <8> vnet_vnetd_connect_forward_socket_begin: [vnet_vnetd.c:
460] ipc_string 64424
10.132.126.144:44483 -> 10.132.113.127:1556
<2>bptestbpcd: EXIT status = 0
12:36:29.170 [25420] <2> bptestbpcd: EXIT status = 0

Kevin Lamb's picture

Also get a clean return from the windows media server also, so it is something with the master server it is having an issue with

Mark_Solutions's picture

How many domains are involved here and is DNS good, if not you will need to use hosts files.

On the Master Server what is the value of the regsitry key Port_BPCD?

It really does look like a name resolution issue or similar - when it compares the masters name with the masters name it cannot do its own lookup and that is where it goes wrong as it cannot extract anything from its own cache.

How many networks does the client have?

Just as a test could you try this ..

In the registry change the Client_Name and Browser to be the short name of the client and the server to be the short name of the Master and Media Servers

Then, also on the client, in its hosts file add the Master, Media and itself in short format

Re-start the client services, do a couple of clear_host_cache commands and see if anything changes

Also make sure AV and Firewalls are turned off while you check this out

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Kevin Lamb's picture

Hi 

There are two domains corp.ad.timeinc.com & ipcmedia.com, none of my other 300 clients have a problem with this only the new 2012 ones

I have changed the entries in the reg to shortname and also in the hosts files, cleared the cache and still get the same response from the Master Server only.....

Thanks for all your help with this by the way

Kevin Lamb's picture

Just to add to the confusion I have ran some bpclntcmd commands on the client and below are the responses which do not show any errors:

C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -server bfbackup
host bfbackup: bfbackup.ipcmedia.com at 10.132.112.22
aliases:     bfbackup.ipcmedia.com     bfbackup     10.132.112.22
 
C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -pn
expecting response from server bfbackup
10.132.113.127 lonbfbvmconn2.corp.ad.timeinc.com 10.132.113.127 64549
 
C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -gethostname
lonbfbvmconn2
 
C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -is_master_server bfbackup
bfbackup is a master server
 

Mark_Solutions's picture

Interesting that the media looks to use the pbx port (1556) whereas the Master uses different (higher) ports

This client doesn't appear under the Client Attributes section of the Masters Host Properties does it?

Just wondering why it doesn't use pbx initially as its first connection type

Does a netstat -a on the client show it using the usual NetBackup ports?

Also, what does a blclntcmd -self give on the client

If a media server can connect then it is something specific with how the Master connects to it, or how the client interprets the connection from the Master

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Kevin Lamb's picture

The client was in the client attribs on the Master Host Properties, I have removed this and retested the connection but the same result.

Netstat looks Ok

  TCP    10.132.113.127:139     LONBFBVMCONN2:0        LISTENING
  TCP    10.132.113.127:1556    bfbackup:34915         TIME_WAIT
  TCP    10.132.113.127:1556    bfbackup:39619         TIME_WAIT
  TCP    10.132.113.127:1556    bfbackup:53650         TIME_WAIT
  TCP    10.132.113.127:1556    bfbackup:56195         TIME_WAIT
 
bpclntcmd looks Ok as well
 
 
C:\Program Files\Veritas\NetBackup\bin>bpclntcmd -self
gethostname() returned: lonbfbvmconn2
host lonbfbvmconn2: lonbfbvmconn2 at 10.132.113.127
aliases:     lonbfbvmconn2     10.132.113.127
getfqdn(lonbfbvmconn2) returned: LONBFBVMCONN2.CORP.AD.TIMEINC.com

Mark_Solutions's picture

Interesting that pbx has four connections all in a time wait state, but maybe just your testing has left these.

What does this show on the client:

netsh int ipv4 show dynamicportrange protocol=tcp

Grabbing at straws now - you probably need to log a case - really needs hands on via webex to go through everything that needs to be seen / done

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Kevin Lamb's picture

Hi Mark,

Ran the netsh:

Protocol tcp Dynamic Port Range
---------------------------------
Start Port      : 49152
Number of Ports : 16384
 
As you say it maybe a case of logging the call.
 
the clients are VM clients but that should not make any difference, I am able to back them up using the vStorage API but what we do with our bigger servers is to use vStorage for non data disks and then a standard backup policy for the extra drives, I can if needed back these up fully by the VM method for now so it is not a show stopper.
 

Mark_Solutions's picture

That is quite a high starting port for tcpip

Not sure if it is worth it but you could try running this from an adminstrative command window and see if it makes any difference:

netsh int ipv4 set dynamicport tcp start=1500 num=63000

That gives the server the maximum possible port range and maximum number of ports (we do that on our Master and Media Servers to improve performance - except msdp ones where we have to change the start port)

I would log a call anyway - keep us updated

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

CRZ's picture

Ignore me at your peril!  wink

You haven't addressed the problem I found yesterday.  I'm STILL seeing that extra space I shouldn't see in your bpcd log for the reverse DNS of your master server's hostname.

To make it easier to spot, I've replaced the space with percent signs in this excerpt from your most recently posted bpcd log:

12:07:39.250 [2680.1320] <2> setup_debug_log: switched debug log file for bpcd
12:07:39.250 [2680.1320] <2> bpcd main: VERBOSE = 0
12:07:39.250 [2680.1320] <2> logparams: C:\Program Files\Veritas\NetBackup\bin\bpcd.exe -standalone
12:07:39.250 [2680.1320] <2> process_requests: offset to GMT 0
12:07:39.266 [2680.1320] <2> logconnections: BPCD ACCEPT FROM 10.132.112.22.55614 TO 10.132.113.127.1556 fd = 292
12:07:39.266 [2680.1320] <2> process_requests: setup_sockopts complete
12:07:39.266 [2680.1320] <2> bpcd peer_hostname: Connection from host bfbackup.ipcmedia.com% (10.132.112.22) port 55614
12:07:39.266 [2680.1320] <2> bpcd valid_server: comparing bfbackup.ipcmedia.com and bfbackup.ipcmedia.com%
12:07:39.266 [2680.1320] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1585] in failed file cache ERR=11001 NAME=bfbackup.ipcmedia.com% SVC=NULL
12:07:39.266 [2680.1320] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1275] vnet_cached_getaddrinfo_and_update() failed 6 0x6
12:07:39.266 [2680.1320] <8> vnet_same_host_and_update: [vnet_addrinfo.c:2848] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME2=bfbackup.ipcmedia.com%
12:07:39.266 [2680.1320] <2> bpcd valid_server: comparing lonbfbnbumedia1.corp.ad.timeinc.com and bfbackup.ipcmedia.com%
12:07:39.266 [2680.1320] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1532] in failed cache ERR=11001 NAME=bfbackup.ipcmedia.com% SVC=NULL
12:07:39.266 [2680.1320] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1275] vnet_cached_getaddrinfo_and_update() failed 6 0x6
12:07:39.266 [2680.1320] <8> vnet_same_host_and_update: [vnet_addrinfo.c:2848] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME2=bfbackup.ipcmedia.com%
12:07:39.266 [2680.1320] <2> bpcd valid_server: comparing lonbfbreddev.ipcmedia.com and bfbackup.ipcmedia.com%
12:07:39.266 [2680.1320] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1532] in failed cache ERR=11001 NAME=bfbackup.ipcmedia.com% SVC=NULL
12:07:39.266 [2680.1320] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1275] vnet_cached_getaddrinfo_and_update() failed 6 0x6
12:07:39.266 [2680.1320] <8> vnet_same_host_and_update: [vnet_addrinfo.c:2848] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME2=bfbackup.ipcmedia.com%
12:07:39.266 [2680.1320] <4> bpcd valid_server: bfbackup.ipcmedia.com% is not a master server
12:07:39.266 [2680.1320] <16> bpcd valid_server: bfbackup.ipcmedia.com% is not a media server either
12:07:39.266 [2680.1320] <8> vnet_cached_getaddrinfo_and_update: [vnet_addrinfo.c:1532] in failed cache ERR=11001 NAME=bfbackup.ipcmedia.com% SVC=NULL
12:07:39.266 [2680.1320] <8> vnet_cached_getaddrinfo: [vnet_addrinfo.c:1275] vnet_cached_getaddrinfo_and_update() failed 6 0x6
12:07:39.266 [2680.1320] <8> vnet_is_local_host: [vnet_addrinfo.c:327] vnet_cached_getaddrinfo() failed STAT=6 RV=11001 NAME=bfbackup.ipcmedia.com%
12:07:39.266 [2680.1320] <2> process_requests: output socket port number = 1

You can also see from this bpcd log that OTHER hosts do not have the extra space after the hostname - for example...

12:06:45.848 [2216.2056] <2> bpcd peer_hostname: Connection from host LONBFBVMCONN2.CORP.AD.TIMEINC.com (10.132.113.127) port 53900

...so it's something unique to JUST your master.

Can you explain why your client's reverse DNS for 10.132.112.22 is appending some invisible space to the FQDN of your master when the reverse DNS for 10.132.113.127 doesn't append that space to *its* FQDN?

The problem with invisible space is...you might not see it when you're doing test DNS lookups, because it's invisible.  :)


bit.ly/770LBN | APPLBN | 761LBN

SOLUTION
Kevin Lamb's picture

Hi Chris,

Would I ignore you??? cheeky

This is a strange one as the DNS is used by all our hosts and none have this issue, only the 2012 server, so is 2012 more sensitive than other servers? I will get our network guys to check this out and let you know.

One thing that is strange is that even if I put the master server in the local hosts file which should in effect override the DNS I still get the same issue.

Going to take another look around the system today with a clear head and see what I can find.

Kev

Kevin Lamb's picture

Embarrased or what, we have located the original issue with the hosts file, this was saved as a .txt file so the server was ignoring this, once this was corrected we could then connect.

Still unsure as to why it is seeing the extra space in DNS though, the QIP entry has been checked and looks good so unless this is in the DNS cache somewhere we are unsure but as I said all our other servers use the same DNS entries without error.

I will continue to look into this.

Thanks for all you help 

SOLUTION