globalDB hostname has not been set (MM status 4)

Matts,

We need to do some troubleshooting here.

run this command /usr/openv/netbackup/bin/admincmd/nbemmcmd -listhosts

Do you see your server listed here as

media  cntpadb2a ?

If No, then your server is not registered as a Media server and do not manually try to register, we'll have netbackup automatically do this.

On the Media server, make sure the bp.conf has the SERVER, CLIENT_NAME, and EMMSERVER entries and all correct Ip address have been populated in /etc/hosts or you might have them in DNS.

Now, If everything is ok, restarting the services on the Media server should automatically register the server as Media server in the EMM.

If this part is done we can move to the tape config.

You can run the following command on the Media server to automatically configure tape devices

/usr/openv/volmgr/bin/tpautoconf -a

If this command fails, let me know, we can troubleshoot, If it completes then we need to start the Media Manger device daemon

/usr/openv/volmgr/bin/ltid

after which you can run the following command to see if tape devices are discovered and configured.

/usr/openv/volmgr/bin/vmoprcmd -d ds

Let me know the outcome.

/usr/openv/volmgr/bin/tpautoconf -a

gave the following error:

 Failed opening the device mappings file

Thanks!

Did you add the media server entry to the bp.conf on the master

SERVER = <media server hostname> ?

Also, from the media server can you see you can telnet on port 1556

#telnet <master server hostname> 1556

from the master server

#telnet <media server hostname > 1556

From the media server i CANNOT telnet on that port.  But from the master to media i can.

I can telenet on bpcd port both ways.

I have never installed a media server threw a firewall so i imagine this is the issue?

Let me know, already got the firewall admins working on that port.

Thanks again!

Starting 6.x all we need for a media server working is the following ports

vnetd - 13724

bpcd  - 13782

pbx -1556

we need bi-directional communication between the master and media servers.

Ok all those ports are now available both directions and i still get the error :(

Anything else you can think of?

/usr/openv/volmgr/bin/tpautoconf -a

Still returns:

 Failed opening the device mappings file

Whats odd is i run that same command on the other nodes in this setup (which all should be the same) i get:

  /usr/openv/volmgr/bin/tpautoconf: symbol lookup error: /usr/openv/lib/libVnbatST.so: undefined symbol: ovgetlocale

PBX doesnt need to be installed for a SAN media server does it?

edit: nevermind it is installed

Matts,

ICS needs to be installed on all servers running as Media servers as they communicate with the EMM.

can we stop and start the services on media server including the pbx, as the pbx picks up the config when it starts and we need to reload the addresses that are used for communication.

/usr/openv/netbackup/bin/goodies/netbackup stop

/opt/VRTSpbx/bin/vxpbx_exchanged stop

 now lets restart

 pbx has to be started before you start netbackup.

/opt/VRTSpbx/bin/vxpbx_exchanged start

/usr/openv/netbackup/bin/goodies/netbackup start

I have been doing those steps for a bit now... seems i have 2 problems.

First node which has a fresh install of the netbackup software gets this:

[root@cntpadb2a ~]# /usr/openv/volmgr/bin/tpautoconf -a
Failed opening the device mappings file

Other 5 nodes get this:

  [root@cntpadb2b admincmd]# /usr/openv/volmgr/bin/tpautoconf -a
/usr/openv/volmgr/bin/tpautoconf: symbol lookup error: /usr/openv/lib/libVnbatST.so: undefined symbol: ovgetlocale

I think the issue between the 2 is a license issue.  I have a working Oracle rac with each node as a media server and we are using a different license out of our "gold" keys.  So i put that key on the first node and it seems to work better.  For instance when i start the other 5 nodes i get the following:

[root@cntpadb2b admincmd]# /usr/openv/netbackup/bin/goodies/netbackup startNetBackup Database Server started./usr/openv/netbackup/bin/nbnos: symbol lookup error: /usr/openv/lib/libVnbat.so: undefined symbol: ovgetlocaleNetBackup Notification Service started./usr/openv/netbackup/bin/nbemm: symbol lookup error: /usr/openv/lib/libVnbat.so: undefined symbol: ovgetlocaleNetBackup Enterprise Media Manager started./usr/openv/netbackup/bin/nbrb: symbol lookup error: /usr/openv/lib/libVnbat.so: undefined symbol: ovgetlocaleNetBackup Resource Broker started.Rebuilding device nodes.Media Manager daemons started.NetBackup request daemon started./usr/openv/netbackup/bin/bpcompatd: symbol lookup error: /usr/openv/lib/libVnbatST.so: undefined symbol: ovgetlocaleNetBackup compatibility daemon started./usr/openv/netbackup/bin/nbjm: symbol lookup error: /usr/openv/lib/libVnbat.so: undefined symbol: ovgetlocaleNetBackup Job Manager started./usr/openv/netbackup/bin/nbpem: symbol lookup error: /usr/openv/lib/libVnbat.so: undefined symbol: ovgetlocaleNetBackup Policy Execution Manager started./usr/openv/netbackup/bin/nbsl: symbol lookup error: /usr/openv/lib/libVnbat.so: undefined symbol: ovgetlocaleNetBackup Service Layer started./usr/openv/netbackup/bin/nbsvcmon: symbol lookup error: /usr/openv/lib/libVnbat.so: undefined symbol: ovgetlocaleNetBackup Service Monitor started.[root@cntpadb2b admincmd]# /usr/openv/volmgr/bin/tpautoconf -a/usr/openv/volmgr/bin/tpautoconf: symbol lookup error: /usr/openv/lib/libVnbatST.so: undefined symbol: ovgetlocale

I also get the following on the first node from the following commands:

[root@cntpadb2a ~]# /usr/openv/volmgr/bin/tpext -get_dev_mappings_ver
Segmentation fault


Whereas on my other RAC cluster which is functioning i get this:

[root@tbstpdb23a admincmd]# /usr/openv/volmgr/bin/tpext -get_dev_mappings_ver
device mappings version in the EMM database is 1.49.12.10
device mappings version from the local file is 1.49.12.10
Local device mappings file is up-to-date

Matts,

Lets focus only on cntpadb2a

- Can you stop and restart the pbx and netbackup services ?

- can you telnet on port 1556 from the cntpadb2a server to your master server?

Now we will enable verbose logging

touch /usr/openv/volmgr/vm.conf ( if the file doesn't exist)

echo "VERBOSE = 5" >> /usr/openv/volmgr/vm.conf

mkdir /usr/openv/volmgr/debug/tpcommand ( if the directory doesn't exist)

Can you now run the command /usr/openv/volmgr/bin/tpautoconf -a ?

And paste the lines from the log file generated from /usr/openv/volmgr/debug/tpcommand/log.101608

Yes to your first 2 questions.  I even had my firewall admin make sure there was nothing being dropped between the master and media servers.

Here is the log file:

[root@cntpadb2a tpcommand]# cat log.101608
19:54:04.889 [8998] <4> tpautoconf_main: /usr/openv/volmgr/bin/tpautoconf -a
19:54:04.890 [8998] <2> mm_getnodename: cached_hostname cntpadb2a.corp.local, cached_method 3
19:54:04.913 [8998] <2> mm_getnodename:  (3) hostname cntpadb2a (from mm_master_config.mm_server_name)
19:54:04.913 [8998] <4> InitThisHostName: ThisHost is cntpadb2a
19:54:04.913 [8998] <16> MmDeviceMappings::InitializeMappings
        : <MmDeviceMappings.cpp:222> failed to open mappings file. errno = 2
19:54:04.913 [8998] <16> tpautoconf_main: Unable to open the device mappings table, status = 1

 I found this link before i eft work, what are your thoughts on this?  Thanks again for all your help!

http://seer.entsupport.symantec.com/docs/308718.htm

Matts,

Symantec once in two months adds supports to new tape devices. To take advantage of those newly supported tape devices, we need to update the EMM database with the external_types.txt they provide.

Looking at the log, I see you do not have the device_mappings.txt file in the location /usr/openv/var/global.

If don't find a file, can you get it netbackup installation disk or the master server anc copy it to the location /usr/openv/var/global?

We need not load anything into the EMM, can we run now /usr/openv/volmgr/bin/tpautoconf -a?

Now i get a segmentation fault when that runs.  Have stop and started NB and PBX as well.

The log file output is better though:

 21:09:59.572 [14451] <4> tpautoconf_main: /usr/openv/volmgr/bin/tpautoconf -a
21:09:59.572 [14451] <2> mm_getnodename: cached_hostname cntpadb2a.corp.local, cached_method 3
21:09:59.596 [14451] <2> mm_getnodename:  (3) hostname cntpadb2a (from mm_master_config.mm_server_name)
21:09:59.596 [14451] <4> InitThisHostName: ThisHost is cntpadb2a
21:09:59.603 [14451] <4> tpautoconf_main: successfully intialized the device mappings table

Matts,

we cannot do with such little logs, we need to enable verbose logging,

can you run the following commands?

/usr/openv/netbackup/bin/vxlogcfg -p 51216 -o 137 -a -s DebugLevel=3;DiagnosticLevel=6

and now rename the file /usr/openv/volmgr/debug/tpcommand/log.101608 /usr/openv/volmgr/debug/tpcommand/log.101608.old

run the command

/usr/openv/volmgr/bin/tpautoconf -a

and paste the log, we'll take it from there.

I just realized something.  This is a 64bit server and the linux server patch i applied was not the 64bit version.  Could this be causing my problems? 

It will certainly cause problems, that is the reason it is causing segmentation faults.

Can you get that done?

and also if the problem doesn't resolve, we need logging enabled for all these three , ignore my previous message for logging.

vxlogcfg -p 51216 -o 137 -a -s DebugLevel=6;DiagnosticLevel=6

vxlogcfg -p 51216 -o 156 -a -s DebugLevel=6;DiagnosticLevel=6

vxlogcfg -p 51216 -o 143 -a -s DebugLevel=6;DiagnosticLevel=6

Cant believe i missed this.  Uninstalling the other mp and installing the new one now.

Uninstalled all MPs and resintalled with the 64bit server version.

Still getting the same error, segmentation fault.  The logs look the same too :(

23:16:07.849 [18760] <4> tpautoconf_main: /usr/openv/volmgr/bin/tpautoconf -a
23:16:07.849 [18760] <2> mm_getnodename: cached_hostname cntpadb2a.corp.local, cached_method 3
23:16:07.873 [18760] <2> mm_getnodename:  (3) hostname cntpadb2a (from mm_master_config.mm_server_name)
23:16:07.873 [18760] <4> InitThisHostName: ThisHost is cntpadb2a
23:16:07.880 [18760] <4> tpautoconf_main: successfully intialized the device mappings table

Matts,

We need additional logging now... can you run the following commands and then run tpautoconf -a?

vxlogcfg -p 51216 -o 137 -a -s DebugLevel=6;DiagnosticLevel=6

vxlogcfg -p 51216 -o 156 -a -s DebugLevel=6;DiagnosticLevel=6

vxlogcfg -p 51216 -o 143 -a -s DebugLevel=6;DiagnosticLevel=6

Thats the thing, i ran those commands and they were succesfull.  But not extra logging was turned on in that log.  Even deleted the log as your suggested earlier and still the same logging level.

If that is the case, your server is core dumping(segmentation fault) even before it reaches the EMM server.

Can you look for core files? look in the root ls -l /core

I guess its Installing the wrong patches might have caused the problem. How did we reinstall?

Can we go for reinstall on one of the media servers, clean, you need to remove the package first SYMCnetbp?

That directory does not exist.  It should be at / right?

I am not sure where it might core dump on your box

but you can search

find / -name core* -exec ls -l {} \;

Matts,

Where do we stand on this problem?

Can you uninstall all the installed packages and reinstall the correct ones?

The libraries are the suspect here.

While speaking with my coworker we found that we are having some serious issues with Storage Foundation on these nodes.  2 of our shared volumes are not coming online.  We are on hold with Symantec now to resolve that.  We will mention this issue... maybe the 2 are connected?

Good news is though i have taken this time to mount the tape drive threw linux and write to a tape.  We were concerned that HP virtual connect may have been a problem as that is the only difference we can find between this RAC and our long functioning RAC. The new RAC are on HP blades and the old one are stand alone HP servers...  Will keep you posted. 

Thanks for all the help again, i really appreciate it. :)

Matt

Good catch! could be they are connected.

Sure, thanks for the update.

All the best! 

Just and update on this problem.

The Storage problem was not related to the netbackup problem.  That issue was related to HP Virtual Connect for Blade Chassis and one of the Storage Arrays, which is a Hitachi 9570.  We moved the volumes to our AMS 1000 and the issue cleared.

Unfortunetly netbackup was still not starting any services.  2 days of troubleshooting and we learned the this issue is an incompatibility issue with Storage Foundation 5.0 MP3.  I have 3 Oracle RAC's 2 production and one for testing all with the same Linux OS and Kernel. The RAC with MP2 is working fine and when i went to install netbackup media server on our test servers we saw the exact same issue.

Symantec also just emailed and said they were able to recreate this issue in their lab. So they are now working on a fix.

Luckily we all agreed that 6 nodes for this RAC was overkill so we removed one, uninstalled Storage Foundation and made it a media server.  Had no problems setting up that media server.

So for all you googler's!  Storage Foundation Cluster File Systerm for Oracle RAC version 5.0MP3 does not work with Netbackup 6.0 SAN Media/Media Servers.