Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Netbackup 7.5 upgrade failed

Created: 18 Jun 2013 • Updated: 08 Sep 2013 | 21 comments
This issue has been solved. See solution.

Hi All,

we have 7.1 master server(Solaris 10), No media server. I was in middle of upgrade, everything went well until i observed a message

EMM interface initialization failed, status = 77

After install script completed i am not able to run any commands related to emm

how ever nbdb_ping works fine

Database [NBDB] is alive and well on server [NB_inblruxbkup01].
 

Attached install trace file

 

 

Operating Systems:

Comments 21 CommentsJump to latest comment

RamNagalla's picture

looks like your attachment is missing... 

Will Restore's picture

pbx service may not be running so start it up:

 

# /opt/VRTSpbx/bin/vxpbx_exchanged start

 

Will Restore -- where there is a Will there is a way

Jaime_Vazquez's picture

Ensure all of the interfaces on th Master have valid information for resolution.

 

If you have more than 110 VLANs defined and configured there is aknown issue for that.

Pawan_Nagra_777's picture

Hi,

Please check the vm.conf file under in /usr/openv/volmgr/

 

MM_SERVER_NAME = Master server Name

AUTO_PATH_CORRECTION=YES

 

Regards, 

Pawan

 

Regards,

Pawan Kumar

Amit Karia's picture

Hi All

Attached install trace file

 

vm.conf

MM_SERVER_NAME = inblruxbkup01.lehman.com
MEDIA_ID_BARCODE_CHARS = 1 8 1:2:3:4:5:6
VERBOSE

 

There are couple of Interfaces in master server 1) is production and 2) is management IP

 

i checked pbx was started during installation

AttachmentSize
install_trace.22756.doc 73.23 KB
Amit Karia's picture

I also see lot of nbpem processes running

 

 root  7625     1   0 16:04:43 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  8444     1   0 16:08:36 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  7654     1   0 16:04:53 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  8343     1   0 16:08:15 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  8231     1   0 16:07:32 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  7780     1   0 16:05:25 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  7825     1   0 16:05:36 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  8056     1   0 16:07:00 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  8261     1   0 16:07:43 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  8431     1   0 16:08:25 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
 root  8298     1   0 16:07:53 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem

nathanmike's picture

@Amit,

could you please run the command bpp -x and see if the process nbemm is running ?

./nathan

Nathan Mike

LPIC-1 and CLA-11

Symantec Registered Partner

nathan.mike@edotcom.be

www.edotcom.be&l

nathanmike's picture

@Amit

bpps -x

 

./nathan

Nathan Mike

LPIC-1 and CLA-11

Symantec Registered Partner

nathan.mike@edotcom.be

www.edotcom.be&l

Amit Karia's picture

Hi Nathan,

 

inblruxbkup01# bpps -x
NB Processes
------------
    root  5598     1   0 15:57:42 ?           3:53 /usr/openv/netbackup/bin/nbemm
    root  6287     1   0 15:59:08 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6659     1   0 16:00:59 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6672     1   0 16:01:11 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5627     1   0 15:57:44 ?           4:27 /usr/openv/netbackup/bin/nbjm
    root  6849     1   0 16:01:53 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6002     1   0 15:58:04 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5601     1   0 15:57:43 ?           0:37 /usr/openv/netbackup/bin/nbrb
    root  5612     1   0 15:57:43 ?           0:21 /usr/openv/netbackup/bin/bprd
    root  5164     1   0 15:57:24 ?           0:00 /usr/openv/netbackup/bin/vnetd -standalone
    root  6123     1   0 15:58:15 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7685     1   0 16:05:04 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5659     1   0 15:57:48 ?           4:02 /usr/openv/netbackup/bin/nbrmms
    root  6238     1   0 15:58:47 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5630     1   0 15:57:44 ?           0:17 /usr/openv/netbackup/bin/bpdbm
    root  7498     1   0 16:03:59 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5694     1   0 15:57:49 ?           0:01 /usr/openv/netbackup/bin/nbsl
    root  6882     1   0 16:02:04 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5741     1   0 15:57:53 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7022     1   0 16:02:25 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5708     1   0 15:57:50 ?           0:06 /usr/openv/netbackup/bin/nbars
    root  7056     1   0 16:02:36 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7519     1   0 16:04:11 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5167     1   0 15:57:24 ?           0:01 /usr/openv/netbackup/bin/bpcd -standalone
    root  6760     1   0 16:01:21 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  4987     1   0 15:56:32 ?           3:06 /usr/openv/db//bin/NB_dbsrv @/usr/openv/var/global/server.conf @/usr/openv/
    root  5230     1   0 15:57:26 ?           0:09 /usr/openv/netbackup/bin/nbaudit
    root  7689     1   0 16:05:15 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5731     1   0 15:57:53 ?           0:03 /usr/openv/netbackup/bin/nbsvcmon
    root  5618     1   0 15:57:44 ?           0:00 /usr/openv/netbackup/bin/bpcompatd
    root  5631  5630   0 15:57:44 ?           0:19 /usr/openv/netbackup/bin/bpjobd
    root  5655     1   0 15:57:47 ?           0:18 /usr/openv/netbackup/bin/nbim
    root  7605     1   0 16:04:21 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6223     1   0 15:58:36 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5749     1   0 15:57:53 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6209     1   0 15:58:25 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7611     1   0 16:04:32 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8321     1   0 16:08:04 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6338     1   0 15:59:19 ?           0:01 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6256     1   0 15:58:57 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6795     1   0 16:01:32 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8072     1   0 16:07:11 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8055     1   0 16:06:59 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6408     1   0 15:59:30 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5724     1   0 15:57:52 ?           0:02 /usr/openv/netbackup/bin/nbcssc -a NetBackup
    root  6821     1   0 16:01:43 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6884     1   0 16:02:15 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7499     1   0 16:04:00 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  6660     1   0 16:01:00 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  5637     1   0 15:57:46 ?           4:51 /usr/openv/netbackup/bin/nbstserv
    root  8159     1   0 16:07:21 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7625     1   0 16:04:43 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8444     1   0 16:08:36 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7654     1   0 16:04:53 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8343     1   0 16:08:15 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8231     1   0 16:07:32 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7780     1   0 16:05:25 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  7825     1   0 16:05:36 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8056     1   0 16:07:00 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8261     1   0 16:07:43 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8431     1   0 16:08:25 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root  8298     1   0 16:07:53 ?           0:00 /usr/openv/netbackup/bin/nbproxy dblib nbpem
    root 20225     1   0 15:07:51 ?           0:01 /usr/openv/netbackup/bin/admincmd/bpstsinfo -UPDATE

MM Processes
------------
    root  5607     1   0 15:57:43 ?           3:09 /usr/openv/volmgr/bin/ltid
    root  5640     1   0 15:57:46 ?           3:07 vmd -v

Shared Symantec Processes
-------------------------
    root 26174     1   0   Jun 17 ?           0:04 /opt/VRTSpbx/bin/pbx_exchange
inblruxbkup01# bpps -x | grep nbpem | wc
      44     440    4224

 

i wonder why so many nbpem process are running

Pawan_Nagra_777's picture

Hi Amit,

stop the NBU services on master server  and apply the Maintenance Release 4 and check the  

Device mapping file also.

Pawan 

 

Regards,

Pawan Kumar

Jean-Pierre Bailly's picture

I would stop NBU, make sure all processes are stopped, make sur PBX is started, restart NBU and see in what situtaion you find yourself. Seems only 1 policy was converted, how many polciies do you have on this system ?

I would also in paralell open a ticket with Symantec Support.

 

Amit Karia's picture

This is our fresh test setup which was initially running with 7.1 and i wanted to get some idea about 7.5 upgrade, i read some where that now imagedb is part of NBDB which was not the case with 7.1, how ever i see lot of errors like

inblruxbkup01# bperror -hoursago 4 -U
    TIME            SERVER/CLIENT                      TEXT
06/19/2013 13:43:47 unknown -  nbjm waiting - nbrb is not running
06/19/2013 13:53:50 unknown -  nbjm waiting - nbrb is not running
06/19/2013 14:03:53 unknown -  nbjm waiting - nbrb is not running
06/19/2013 14:13:56 unknown -  nbjm waiting - nbrb is not running
06/19/2013 14:23:58 unknown -  nbjm waiting - nbrb is not running
06/19/2013 14:34:01 unknown -  nbjm waiting - nbrb is not running
06/19/2013 14:44:03 unknown -  nbjm waiting - nbrb is not running
06/19/2013 14:54:06 unknown -  nbjm waiting - nbrb is not running
06/19/2013 15:04:09 unknown -  nbjm waiting - nbrb is not running
06/19/2013 15:14:11 unknown -  nbjm waiting - nbrb is not running
06/19/2013 15:24:14 unknown -  nbjm waiting - nbrb is not running

and

06/19/2013 15:49:29 inblruxbkup01.lehman.com -  connection to
                    nbevtmgr::VRTS_NBU_PEM_Channel on host
                    inblruxbkup01.lehman.com failed after 7 attempts

that is the reason i guess for emm not responding , i also noticed  vxdbms.conf is still with older date (before upgrade) looks like nbu has not been upgraded properly.

 

Yes there is only 1 policy(catalog backup)

 

 

nathanmike's picture

@Amit,

Could you please clarify exactly your upgrade scenario ? i am a little bit confused

What is your current major version ? 7.1 ? 7.5 ?

what is you actual patch version ? 7.1.0.3 ? 7.5.0.4? 7.5.0.5?

Please run the following command:

bpclntcmd -get_local_server_patch_version

Nathan Mike

LPIC-1 and CLA-11

Symantec Registered Partner

nathan.mike@edotcom.be

www.edotcom.be&l

Amit Karia's picture

inblruxbkup01# bpclntcmd -get_local_server_patch_version
7.5.0.0
 

it is 7.5 but i think not fully upgraded or with some issues

 

nathanmike's picture

@Amit,

ok. You are using the version 7.5 which mean at this time your system is not patched at all.

As you mentionned in the beginning, you are using a Solaris 10 master server right?

Is it a cluster node or standalone ?

If it's a Cluster node , there are some particular steps that you need to follow. You can consult you netbackup installation guide for UNIX and go to the cluster section. You will find more details about it.

if it's a standalone, then extract the archive and apply the patch

I assume that your images have been migrated successfully and your catalog database is ok (you can run nbcc tools for checking the integrity of your database.

If you have upload the binaries to your master server, you can find all relevant .pdf documentation under the following directory:

<path-to-you-upload-folder>/NetBackup_7.5_LinuxR_x86_64/Doc

mine is as follow:

[root@master]# ls
favicon.ico                                     NetBackup_AdminGuide_Sybase.pdf
NetBackup_3rdpartyattributions.pdf              NetBackup_AdminGuide_Vault.pdf
NetBackup_AdminGuide_BMR.pdf                    NetBackup_AdminGuide_VMware.pdf
NetBackup_AdminGuide_Cloud.pdf                  NetBackup_AdvDisk_Guide.pdf
NetBackup_AdminGuide_ClusteredMasterServer.pdf  NetBackup_BAR_GS_Guide.pdf
NetBackup_AdminGuide_DB2.pdf                    NetBackup_Commands.pdf
NetBackup_AdminGuide_EntVault.pdf               NetBackup_Dedupe_Guide.pdf
NetBackup_AdminGuide_HighAvailability.pdf       NetBackup_DeviceConfig_Guide.pdf
NetBackup_AdminGuide_Hyper-V.pdf                NetBackup_GettingStarted_Guide.pdf
NetBackup_AdminGuideII_UNIXServer.pdf           NetBackup_Install_UNIX.pdf
NetBackup_AdminGuideII_WinServer.pdf            NetBackup_Install_Win.pdf
NetBackup_AdminGuide_Informix.pdf               NetBackup_LiveUpdate_Guide.pdf
NetBackup_AdminGuideI_UNIXServer.pdf            NetBackupManuals.html
NetBackup_AdminGuideI_WinServer.pdf             NetBackup_OperGuide_Vault.pdf
NetBackup_AdminGuide_LotusNotes.pdf             NetBackup_OST_Disk_Guide.pdf
NetBackup_AdminGuide_MSExchg_Win.pdf            NetBackup_RefGuide_StatusCodes.pdf
NetBackup_AdminGuide_MSSQL_Win.pdf              NetBackup_Release_Notes.pdf
NetBackup_AdminGuide_NDMP.pdf                   NetBackup_RepDirector_Guide.pdf
NetBackup_AdminGuide_Oracle.pdf                 NetBackup_SANClient_Guide.pdf
NetBackup_AdminGuide_SAP.pdf                    NetBackup_SecEncryp_Guide.pdf
NetBackup_AdminGuide_Search.pdf                 NetBackup_Troubleshoot_Guide.pdf
NetBackup_AdminGuide_SharePoint.pdf             symc_logo.gif
NetBackup_AdminGuide_SnapshotClient.pdf
[root@master]# pwd
/data/netbackup/pkgs/7.5/NetBackup_7.5_LinuxR_x86_64/Doc
[root@master]#
 

Before patching:

1. reboot completly

2. make sure all process are up with bpps -x

3. Try connect to admin console

4. run NBCC tools to check the integrity of your catalog

5. check EMM integrity with the following command:

nbemmcmd -getemmserver

nbemmcmd -listmedia -allrecords -brief

nbemmcmd -listhosts

If all of the above sounds good, try to patch and tack the logs ;-)

Nathan Mike

LPIC-1 and CLA-11

Symantec Registered Partner

nathan.mike@edotcom.be

www.edotcom.be&l

Jaime_Vazquez's picture

No to belabor a point, but for this specific error condition, I think this forum is not the appropriate place to do triage and look for resolution of your problem.  The NetBackup support team should be handling this directly.  You would be better served if you open a support case for this and will get much better assistance to resolve this matter. I am a Back Line support engineer and I am currently working a customer situation almost exaclty as yours.  It is not a trivial matter.

Amit Karia's picture

I agree Jaime, since this is not a prod environment i was not keen to open up a case but looks like i will have to do that

 

@nathan , this is a standalone server, problem i see here is , although emm services are running none of emm commands, (nbemmcmd -listhosts, nbemmcmd -listmedia -mediaid) are respoding , I even tried to check consistency with bpdbm, but it failed.. Looks like i will have to contact support

 

inblruxbkup01# bpdbm -consistency 2
Checking NetBackup database consistency
NetBackup 7.5 2012020723
 Failed to load the media db: cannot connect on socket (25)

check images for clients

I wil keep all of you updated

nathanmike's picture

@Amit,

Just for information:

If i understand correctly, you are migrating from 7.1 to 7.5?

What was the previous patch level before migrating to 7.5 (eg. 7.1.0.3) ?

Did you follow all the steps and pre-req before migrating from 7.1 to 7.5 (deactivate policies, media, SLP, STU, no jobs running ...)?

Sometimes, if you miss one of those pre-req, it can lead to such situation. What Jaime recommend you is correct, this can be perfectly handle by Symantec support.

Good luck ;-)

Nathan Mike

LPIC-1 and CLA-11

Symantec Registered Partner

nathan.mike@edotcom.be

www.edotcom.be&l

Mark_Solutions's picture

If this is a test then remember there are real issue when you upgrade to 7.5

You need to run the upgrade then take NBU down straight away and patch to 7.5.0.5

Once this is done the catalog migration to EMM wil start and only once that has completed will NBU start to fully function

Hope this helps

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

SOLUTION