Video Screencast Help

Import failed for backup id car-sql1_1353898826 with status 1535 (AIR)

Created: 03 Dec 2012 • Updated: 21 Feb 2013 | 24 comments
Spartacus81's picture
This issue has been solved. See solution.

Friends,

Source Master 7.5.0.3 + 8 Media servers

Target Master 7.5.0.3 + 8 Media servers

IMPORT job on target master finishing partially and am getting the following error..

12/03/2012 08:55:38 - begin Import
12/03/2012 08:55:38 - Info bpdm (pid=30373) started
12/03/2012 08:55:38 - started process bpdm (pid=30373)
12/03/2012 08:55:38 - requesting resource @aaaak
12/03/2012 08:55:38 - granted resource MediaID=@aaaak;DiskVolume=PureDiskVolume;DiskPool=gsgs2nbmed04_Pool4;Path=PureDiskVolume;StorageServer=gsgs2nbmed04;MediaServer=gsgs2nbmed04
12/03/2012 08:55:38 - started process RUNCMD (pid=23087)
12/03/2012 08:55:39 - Info bpdm (pid=30373) reading backup image
12/03/2012 08:55:41 - Info bpdm (pid=30373) origin master server gssuanbmas01, backup id car-sql1_1353898826
12/03/2012 08:55:41 - Error bpdm (pid=30373) received error code 1535 from db_end() after writing image metadata information to bpdbm
12/03/2012 08:55:41 - Error bpimport (pid=23087) Import of replica image, backup id car-sql1_1353898826, Import failed because the imported image backup id conflicts with an existing image.
12/03/2012 08:55:41 - Import failed for backup id car-sql1_1353898826 with status 1535

i have followed the following tech note which tells to remove the duplicate images from the target Storage which is my case the target media server as i cant see the images in question on the remote master or within the catalogue but no joy..

http://www.symantec.com/business/support/index?page=content&id=HOWTO72896

also, only 1 media server is effected is rest of them replicating okay..

looking at the dedup database area i can see all the images files which IMPORT job trying to import again. i have moved all these images to /tmp location but still no joy..

-rw-r----- 1 root root 166 Nov 26 03:14 car-sql1_1353898826_C1_F1.hdr
-rw-r----- 1 root root 168 Nov 26 03:14 car-sql1_1353898826_C1_F1.img
-rw-r----- 1 root root 161 Nov 26 03:14 car-sql1_1353898826_C1_F1.info
-rw-r----- 1 root root 164 Nov 26 03:14 car-sql1_1353898826_C1_F1.map
-rw-r----- 1 root root 162 Nov 26 03:14 car-sql1_1353898826_C1_HDR.img
-rw-r----- 1 root root 161 Nov 26 03:14 car-sql1_1353898826_C1_HDR.info
-rw-r----- 1 root root 163 Nov 26 03:14 car-sql1_1353898826_C1_IM.img
-rw-r----- 1 root root 162 Nov 26 03:14 car-sql1_1353898826_C1_IM.info
-rw-r----- 1 root root 165 Nov 26 03:14 car-sql1_1353898826_C1_TIR.img
-rw-r----- 1 root root 161 Nov 26 03:14 car-sql1_1353898826_C1_TIR.info

so can someone confirm where should i be deleting the duplicate images as per the tech note, master or media? on media i can see the images under /databases but on the master i can't see the images under ../db/images/car-sql1 directory?

any suggestions would highly be appreciated.

Comments 24 CommentsJump to latest comment

Nagalla's picture

hi,

 

bpimport log with Verobose 5 could help us,

please post the bpimport log..

Will Restore's picture

check output of this command for clues

nbstlutil -list -backupid car-sql1_1353898826  -U

 

Will Restore -- where there is a Will there is a way

Spartacus81's picture

thanks Nagalla. am using AIR (auto image replication) and this import process is automatic.

thank WR.  ran this command and on the source master and it all appears to be replicated.

gssuanbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil list -backupid car-sql1_1353898826 -U
Image:
 Master Server            : gssuanbmas01
 Backup ID                : car-sql1_1353898826
 Client                   : car-sql1
 Backup Time              : 1353898826 (Mon Nov 26 03:00:26 2012)
 Policy                   : gssuanbmed04_MS_Sites
 Client Type              : 13
 Schedule Type            : 1
 Storage Lifecycle Policy : gssuanbmed04_to_gsgs2nbmed04_2Weeks
 Storage Lifecycle State  : 3 (COMPLETE)
 Time In Process          : 1353899512 (Mon Nov 26 03:11:52 2012)
 Data Classification ID   : (none specified)
 Version Number           : 0
 OriginMasterServer       : (none specified)
 OriginMasterServerID     : 00000000-0000-0000-0000-000000000000
 Import From Replica Time : 0 (Thu Jan  1 01:00:00 1970)
 Required Expiration Date : 0 (Thu Jan  1 01:00:00 1970)
 Created Date Time        : 1353898826 (Mon Nov 26 03:00:26 2012)

 Copy:
   Master Server       : gssuanbmas01
   Backup ID           : car-sql1_1353898826
   Copy Number         : 1
   Copy Type           : 0 (BACKUP)
   Expire Time         : 1355108426 (Mon Dec 10 03:00:26 2012)
   Expire LC Time      : 1355108426 (Mon Dec 10 03:00:26 2012)
   Try To Keep Time    : 1355108426 (Mon Dec 10 03:00:26 2012)
   Residence           : gssuanbmed04_Pool4
   Copy State          : 3 (COMPLETE)
   Job ID              : 12949
   Retention Type      : 0 (FIXED)
   MPX State           : 0 (FALSE)
   RetryCount          : 0
   Last Retry Time     : 1353899385 (Mon Nov 26 03:09:45 2012)
   Source              : 0
   Destination ID      : (none specified)
   Last Retry Time     : 1353899385
   Replica:            : 0
   DataFormat          : 1 (DF_TAR)
   SLP Index           : 1

   Fragment:
     Master Server     : gssuanbmas01
     Backup ID         : car-sql1_1353898826
     Copy Number       : 1
     Fragment Number   : -2147482648
     Resume Count      : 0
     Media ID          : @aaaau
     Media Server      : gssuanbmed04
     Storage Server    : (none specified)
     Media Type        : 0 (DISK)
     Media Sub-Type    : 6 (STSDYNAMIC)
     Fragment State    : 1 (ACTIVE)
     Fragment Size     : 1320
     Delete Header     : 0
     Fragment ID       : @aaaau
     Snap MountHost    : (none specified)
     Media Description : 1;PureDisk;gssuanbmed04;gssuanbmed04_Pool4;PureDiskVolume;0

   Fragment:
     Master Server     : gssuanbmas01
     Backup ID         : car-sql1_1353898826
     Copy Number       : 1
     Fragment Number   : -1
     Resume Count      : 0
     Media ID          : @aaaau
     Media Server      : gssuanbmed04
     Storage Server    : (none specified)
     Media Type        : 0 (DISK)
     Media Sub-Type    : 6 (STSDYNAMIC)
     Fragment State    : 1 (ACTIVE)
     Fragment Size     : 4204229
     Delete Header     : 0
     Fragment ID       : @aaaau
     Snap MountHost    : (none specified)
     Media Description : 1;PureDisk;gssuanbmed04;gssuanbmed04_Pool4;PureDiskVolume;0

   Fragment:
     Master Server     : gssuanbmas01
     Backup ID         : car-sql1_1353898826
     Copy Number       : 1
     Fragment Number   : 1
     Resume Count      : 0
     Media ID          : @aaaau
     Media Server      : gssuanbmed04
     Storage Server    : (none specified)
     Media Type        : 0 (DISK)
     Media Sub-Type    : 6 (STSDYNAMIC)
     Fragment State    : 1 (ACTIVE)
     Fragment Size     : 6035368960
     Delete Header     : 0
     Fragment ID       : @aaaau
     Snap MountHost    : (none specified)
     Media Description : 1;PureDisk;gssuanbmed04;gssuanbmed04_Pool4;PureDiskVolume;0

 Copy:
   Master Server       : gssuanbmas01
   Backup ID           : car-sql1_1353898826
   Copy Number         : 102
   Copy Type           : 3 (REPLICATE_TO_REMOTE_MASTER)
   Expire Time         : 1355108426 (Mon Dec 10 03:00:26 2012)
   Expire LC Time      : 1355108426 (Mon Dec 10 03:00:26 2012)
   Try To Keep Time    : 1355108426 (Mon Dec 10 03:00:26 2012)
   Residence           : Remote Master
   Copy State          : 3 (COMPLETE)
   Job ID              : 12982
   Retention Type      : 0 (FIXED)
   MPX State           : 0 (FALSE)
   RetryCount          : 1
   Last Retry Time     : 1353899669 (Mon Nov 26 03:14:29 2012)
   Source              : 1
   Target ID           : (none specified)
   Last Retry Time     : 1353899669
   Replica:            : 0
   DataFormat          : 0 (DF_UNDEFINED)
   SLP Index           : 2

gssuanbmas01:~ #

 

any other suggesstion?

Bugs comes in through open windows!

Spartacus81's picture

hello friends, any comments?

Bugs comes in through open windows!

Spartacus81's picture

its been two weeks and i have no joy with this thread... right, the image that had been replicated but never got imported has expired from the source domain and there is no clue of them in the target domain but when i issue the following command..

gsgs2nbmas01:/ # /usr/openv/netbackup/bin/admincmd/nbstlutil pendimplist
V7.5.0
I gsgs2nbmas01 car-sql1_1353898826 car-sql1 1353898826 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909044
V7.5.0 C gsgs2nbmas01 car-sql1_1353898826 1 0 0 gsgs2nbmed04_Pool4 1 14579 3 0 0 *NULL* 0 1355305831 0 201 4 0 0 1
V7.5.0 F gsgs2nbmas01 car-sql1_1353898826 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 car-sql1_1353898826_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 ph1-fs1_1353898997 ph1-fs1 1353898997 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909054
V7.5.0 C gsgs2nbmas01 ph1-fs1_1353898997 1 0 0 gsgs2nbmed04_Pool4 1 14579 3 0 0 *NULL* 0 1355305831 0 201 4 0 0 1
V7.5.0 F gsgs2nbmas01 ph1-fs1_1353898997 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 ph1-fs1_1353898997_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 ph1-fs1_1353899165 ph1-fs1 1353899165 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909068
V7.5.0 C gsgs2nbmas01 ph1-fs1_1353899165 1 0 0 gsgs2nbmed04_Pool4 1 14579 3 0 0 *NULL* 0 1355305831 0 201 4 0 0 1
V7.5.0 F gsgs2nbmas01 ph1-fs1_1353899165 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 ph1-fs1_1353899165_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mesnsql01_1353899222 mesnsql01 1353899222 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909084
V7.5.0 C gsgs2nbmas01 mesnsql01_1353899222 1 0 0 gsgs2nbmed04_Pool4 1 14579 3 0 0 *NULL* 0 1355305831 0 201 4 0 0 1
V7.5.0 F gsgs2nbmas01 mesnsql01_1353899222 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mesnsql01_1353899222_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mesnsql01_1353899242 mesnsql01 1353899242 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909104
V7.5.0 C gsgs2nbmas01 mesnsql01_1353899242 1 0 0 gsgs2nbmed04_Pool4 1 14579 3 0 0 *NULL* 0 1355305831 0 201 4 0 0 1
V7.5.0 F gsgs2nbmas01 mesnsql01_1353899242 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mesnsql01_1353899242_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mspc512fpx01_1353899436 mspc512fpx01 1353899436 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909136
V7.5.0 C gsgs2nbmas01 mspc512fpx01_1353899436 1 0 0 gsgs2nbmed04_Pool4 1 14579 3 0 0 *NULL* 0 1355305831 0 201 4 0 0 1
V7.5.0 F gsgs2nbmas01 mspc512fpx01_1353899436 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mspc512fpx01_1353899436_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 gssuanbmas01_1355327840 gssuanbmas01 1355327840 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 1 0 *NULL* 2 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1355330028
V7.5.0 C gsgs2nbmas01 gssuanbmas01_1355327840 1 0 0 gsgs2nbmed04_Pool4 1 0 3 0 0 *NULL* 0 0 0 0 4 0 0 1
V7.5.0 F gsgs2nbmas01 gssuanbmas01_1355327840 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 gssuanbmas01_1355327840_C1_IM *NULL* gsgs2nbmed04
 

it list me the above images which are now orphaned.. in SLP there is a cancel parameter as seen below.

-- cancel pending operations on selected image copies
   cancel [-wait | -force]
          [[-lifecycle <name> [-destination <name>] [-version <number>] [-nowarn]] |
           -backupid <id_value>]

am running following command which returns no error but doesnt actually cancel the operation on the particular image...

gsgs2nbmas01:/ # /usr/openv/netbackup/bin/admincmd/nbstlutil cancel -backupid car-sql1_1353898826
 

can someone please suggest anything here as the SLP in question completing partially and which is no harm but i dont like partials..

any suggestion will highly be appreciated.

 

 

 

Bugs comes in through open windows!

Tomer Gurantz's picture

Have you tried running "nbstlutil pendimplist" in the target domain, which should show pending import copies. (In theory "nbstlutil repllist" could be run in the source domain to show that the replication was completed successfully, but I think you are far past that stage).

Import is handled by a resource library called Import Manager of nbstserv process, and is initiated by information provided by the STS Event Manager (debug log OID 395) which is a service library of nbrmms (which manages much of the OST-based disk storage, including dedupe), so logging those are an option, but I wouldn't be able to decipher. You'd need to talk to support.

 

Also, the LIFECYCLE_PARAMETERS file (see documentation) can be used to customize much of how SLPs work, and specifically there is a parameter which is by default set to REPLICA_METADATA_CLEANUP_TIMER = 0. The definition: Days after which to delete unimported import records. If you changed this to 14, for example, than unimported records would be deleted automatically after 2 weeks. Not sure if this is what you'd want. Note that this setting would need to be placed in the target domain (the one doing the imports)

Also, you mention running "cancel" which cancels operations, but perhaps an import isn't considered in this command option, just duplication, backup and replication jobs? Not sure, I don't have enough experience with SLP imports in this light.

 

Senior Principal Technical Education Consultant with Symantec Corporation

Spartacus81's picture

Thanks Tomer. I thought am very good at SLP's but am clearly not. Yes i tried the pendimplist already and it does show the images.

gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil pendimplist
V7.5.0 I gsgs2nbmas01 car-sql1_1353898826 car-sql1 1353898826 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909044
V7.5.0 C gsgs2nbmas01 car-sql1_1353898826 1 0 0 gsgs2nbmed04_Pool4 1 26589 3 0 0 *NULL* 0 1358336557 0 632 4 0 0 1
V7.5.0 F gsgs2nbmas01 car-sql1_1353898826 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 car-sql1_1353898826_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 ph1-fs1_1353898997 ph1-fs1 1353898997 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909054
V7.5.0 C gsgs2nbmas01 ph1-fs1_1353898997 1 0 0 gsgs2nbmed04_Pool4 1 26589 3 0 0 *NULL* 0 1358336557 0 632 4 0 0 1
V7.5.0 F gsgs2nbmas01 ph1-fs1_1353898997 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 ph1-fs1_1353898997_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 ph1-fs1_1353899165 ph1-fs1 1353899165 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909068
V7.5.0 C gsgs2nbmas01 ph1-fs1_1353899165 1 0 0 gsgs2nbmed04_Pool4 1 26589 3 0 0 *NULL* 0 1358336557 0 632 4 0 0 1
V7.5.0 F gsgs2nbmas01 ph1-fs1_1353899165 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 ph1-fs1_1353899165_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mesnsql01_1353899222 mesnsql01 1353899222 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909084
V7.5.0 C gsgs2nbmas01 mesnsql01_1353899222 1 0 0 gsgs2nbmed04_Pool4 1 26589 3 0 0 *NULL* 0 1358336557 0 632 4 0 0 1
V7.5.0 F gsgs2nbmas01 mesnsql01_1353899222 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mesnsql01_1353899222_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mesnsql01_1353899242 mesnsql01 1353899242 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909104
V7.5.0 C gsgs2nbmas01 mesnsql01_1353899242 1 0 0 gsgs2nbmed04_Pool4 1 26589 3 0 0 *NULL* 0 1358336557 0 632 4 0 0 1
V7.5.0 F gsgs2nbmas01 mesnsql01_1353899242 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mesnsql01_1353899242_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mspc512fpx01_1353899436 mspc512fpx01 1353899436 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909136
V7.5.0 C gsgs2nbmas01 mspc512fpx01_1353899436 1 0 0 gsgs2nbmed04_Pool4 1 26589 3 0 0 *NULL* 0 1358336557 0 632 4 0 0 1
V7.5.0 F gsgs2nbmas01 mspc512fpx01_1353899436 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mspc512fpx01_1353899436_C1_IM *NULL* gsgs2nbmed04
 

These images have been expired long ago as per their retention but import SLP at target domain still trying to unsuccessfully import them hence finishing patially.

I'll try the lifecycle parameter file to see if that works all i want is target domain (remote master that is doing the import not the media server associated to that images) to forget about these images as they are no longer exist.

i have created the file and set the number to 7. Let see how it goes. will update the thread next week Thursday.

 

Bugs comes in through open windows!

Tomer Gurantz's picture

Yes, let's see. I wonder if you'll see results earlier, after all the 7 days might mean 7 days from when it started the import, not 7 days from today (but I'm not 100% sure about that).

SLP ensures that images do not get deleted until all duplications (and in this case import/replications) are completed, even if the retention has been reached. (In my experience/knowledge)

Senior Principal Technical Education Consultant with Symantec Corporation

Tomer Gurantz's picture

Oh yes, also in this post (https://www-secure.symantec.com/connect/forums/can...) they mention a similar issue and used the "-force" option to cancel the import jobs:

So, add the -force to your prior command:

# /usr/openv/netbackup/bin/admincmd/nbstlutil cancel -force -backupid car-sql1_1353898826

Senior Principal Technical Education Consultant with Symantec Corporation

Spartacus81's picture

No joy tomer.. tried that already its not helping. even though the cancel command doesnt show any error so assuming successful. see result below:

gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil pendimplist
V7.5.0 I gsgs2nbmas01 car-sql1_1353898826 car-sql1 1353898826 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909044
V7.5.0 C gsgs2nbmas01 car-sql1_1353898826 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 car-sql1_1353898826 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 car-sql1_1353898826_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 ph1-fs1_1353898997 ph1-fs1 1353898997 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909054
V7.5.0 C gsgs2nbmas01 ph1-fs1_1353898997 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 ph1-fs1_1353898997 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 ph1-fs1_1353898997_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 ph1-fs1_1353899165 ph1-fs1 1353899165 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909068
V7.5.0 C gsgs2nbmas01 ph1-fs1_1353899165 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 ph1-fs1_1353899165 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 ph1-fs1_1353899165_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mesnsql01_1353899222 mesnsql01 1353899222 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909084
V7.5.0 C gsgs2nbmas01 mesnsql01_1353899222 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 mesnsql01_1353899222 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mesnsql01_1353899222_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mesnsql01_1353899242 mesnsql01 1353899242 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909104
V7.5.0 C gsgs2nbmas01 mesnsql01_1353899242 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 mesnsql01_1353899242 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mesnsql01_1353899242_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mspc512fpx01_1353899436 mspc512fpx01 1353899436 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909136
V7.5.0 C gsgs2nbmas01 mspc512fpx01_1353899436 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 mspc512fpx01_1353899436 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mspc512fpx01_1353899436_C1_IM *NULL* gsgs2nbmed04
gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil cancel -force -backupid car-sql1_1353898826
gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil cancel -force -backupid ph1-fs1_1353898997
gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil cancel -force -backupid ph1-fs1_1353899165
gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil cancel -force -backupid mesnsql01_1353899222
gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil cancel -force -backupid mesnsql01_1353899242
gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil cancel -force -backupid mspc512fpx01_1353899436

gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil pendimplist
V7.5.0 I gsgs2nbmas01 car-sql1_1353898826 car-sql1 1353898826 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909044
V7.5.0 C gsgs2nbmas01 car-sql1_1353898826 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 car-sql1_1353898826 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 car-sql1_1353898826_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 ph1-fs1_1353898997 ph1-fs1 1353898997 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909054
V7.5.0 C gsgs2nbmas01 ph1-fs1_1353898997 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 ph1-fs1_1353898997 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 ph1-fs1_1353898997_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 ph1-fs1_1353899165 ph1-fs1 1353899165 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909068
V7.5.0 C gsgs2nbmas01 ph1-fs1_1353899165 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 ph1-fs1_1353899165 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 ph1-fs1_1353899165_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mesnsql01_1353899222 mesnsql01 1353899222 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909084
V7.5.0 C gsgs2nbmas01 mesnsql01_1353899222 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 mesnsql01_1353899222 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mesnsql01_1353899222_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mesnsql01_1353899242 mesnsql01 1353899242 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909104
V7.5.0 C gsgs2nbmas01 mesnsql01_1353899242 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 mesnsql01_1353899242 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mesnsql01_1353899242_C1_IM *NULL* gsgs2nbmed04
V7.5.0 I gsgs2nbmas01 mspc512fpx01_1353899436 mspc512fpx01 1353899436 *NULL* 0 0 gssuanbmed04_to_gsgs2nbmed04_2Weeks 3 0 *NULL* 1 gssuanbmas01 00000000-0000-0000-0000-000000000000 0 0 1353909136
V7.5.0 C gsgs2nbmas01 mspc512fpx01_1353899436 1 0 0 gsgs2nbmed04_Pool4 1 28395 3 0 0 *NULL* 0 1358758057 0 693 4 0 0 1
V7.5.0 F gsgs2nbmas01 mspc512fpx01_1353899436 1 -2147482648 0 @aaaak gsgs2nbmed04 *NULL* 0 0 1 0 0 mspc512fpx01_1353899436_C1_IM *NULL* gsgs2nbmed04
gsgs2nbmas01:~ #

fingers crossed till thursday :)

 

Bugs comes in through open windows!

Spartacus81's picture

No Joy guys.. the images are still there even after 7 days.. :S and its annoying..
any other suggestionS?

Bugs comes in through open windows!

Tomer Gurantz's picture

Ya, contact support. Seriously, maybe they can suggest something we've missed.

I'm assuming you tried the 'cancel' with the -force operation? I wouldn't expect it to work in this scenario without the -force since you are asking it to do something without double-check type verification (i.e. the image itself isn't around, just a record of it).

Senior Principal Technical Education Consultant with Symantec Corporation

Spartacus81's picture

hmmm support is the only option now left..
yes i did try cancel operation few times with force operation and it appeared to work but then you check pending import list - they are still there...

Bugs comes in through open windows!

Mark_Solutions's picture

Just in case please go onto the appliance trying to import the images - go into maintenance mode and cd to:

/disk/databases/catalog/2/

Take a look to see if the servers in question have more than one directory showing in different cases

i.e. sql1 and SQL1

If they do then you may just need a softlink creating between the two directories to get everything working:

ln -s /disk/databases/catalog/2/sql1 /disk/databases/catalog/SQL1

(at least i think that is the right command - do double check)

 #edit# you may beed a service re-start after creating the softlink

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Spartacus81's picture

Thanks Mark.. but no there are no two directories for this car-sql1 server name...
am planning on rebooting the both masters as they have been up for over 70days to see if that helps...

Bugs comes in through open windows!

Mark_Solutions's picture

Did you ever get this sorted out?

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Spartacus81's picture

sorry nooops mark.. still struggling with it..

i will put my feedback definitely.. can you think of anything in the meantime?

Bugs comes in through open windows!

Mark_Solutions's picture

Not new ideas as yet - trying to really get to the bottom of where everything is stored for AIR as i have a case ongoing at the moment myself with similar issues

I have the feeling that the information may be helpd in the EMM Database but haven't got to the bottom of it yet!

One thing that may help - at least to stop it keep running may be

nbstlutil inactive -backupid car-sql1_1353898826

But again whether this does anything is another matter

Just re-reading the guides and they do point at all of this being hel in a table in EMM so that is really where changes need to be made

This needs support to do a sql script to edit the tables though (catalog backup first me thinks!!)

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Spartacus81's picture

Thanks Mark its never easy especially if one is obsessed with the blue men :)

thanks for the suggestion.. but just to confirm setting that image inactive will not stop the SLP to processing the new images.. i think it wont but just to double check..

Bugs comes in through open windows!

Mark_Solutions's picture

My feeling is that as the command has the option to specify the backupid that it will not affect any others.

You could find a "real" backupid that is due for import (not easy as the things fly through!) and then run

nbstlutil list -U -backupid <backupid> against it and the bad one before and after to see if their states actually change

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Spartacus81's picture

gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil list -U -backupid car-sql1_1353898826
Image:
 Master Server            : gsgs2nbmas01
 Backup ID                : car-sql1_1353898826
 Client                   : car-sql1
 Backup Time              : 1353898826 (Mon Nov 26 03:00:26 2012)
 Policy                   : (none specified)
 Client Type              : 0
 Schedule Type            : 0
 Storage Lifecycle Policy : gssuanbmed04_to_gsgs2nbmed04_2Weeks
 Storage Lifecycle State  : 3 (COMPLETE)
 Time In Process          : 0 (Thu Jan  1 01:00:00 1970)
 Data Classification ID   : (none specified)
 Version Number           : 1
 OriginMasterServer       : gssuanbmas01
 OriginMasterServerID     : 00000000-0000-0000-0000-000000000000
 Import From Replica Time : 0 (Thu Jan  1 01:00:00 1970)
 Required Expiration Date : 0 (Thu Jan  1 01:00:00 1970)
 Created Date Time        : 1353909044 (Mon Nov 26 05:50:44 2012)

 Copy:
   Master Server       : gsgs2nbmas01
   Backup ID           : car-sql1_1353898826
   Copy Number         : 1
   Copy Type           : 4 (IMPORT)
   Expire Time         : 0 (Thu Jan  1 01:00:00 1970)
   Expire LC Time      : 0 (Thu Jan  1 01:00:00 1970)
   Try To Keep Time    : 0 (Thu Jan  1 01:00:00 1970)
   Residence           : gsgs2nbmed04_Pool4
   Copy State          : 1 (NOT_STARTED)
   Job ID              : 41209
   Retention Type      : 3 (REMOTE_EXPIRATION_DATE)
   MPX State           : 0 (FALSE)
   RetryCount          : 1070
   Last Retry Time     : 1361351122 (Wed Feb 20 09:05:22 2013)
   Source              : 0
   Destination ID      : (none specified)
   Last Retry Time     : 1361351122
   Replica:            : 0
   DataFormat          : 0 (DF_UNDEFINED)
   SLP Index           : 1

   Fragment:
     Master Server     : gsgs2nbmas01
     Backup ID         : car-sql1_1353898826
     Copy Number       : 1
     Fragment Number   : -2147482648
     Resume Count      : 0
     Media ID          : @aaaak
     Media Server      : gsgs2nbmed04
     Storage Server    : (none specified)
     Media Type        : 0 (DISK)
     Media Sub-Type    : 0 (DEFAULT)
     Fragment State    : 1 (ACTIVE)
     Fragment Size     : 0
     Delete Header     : 0
     Fragment ID       : car-sql1_1353898826_C1_IM
     Snap MountHost    : (none specified)
     Media Description : gsgs2nbmed04

gsgs2nbmas01:~ # /usr/openv/netbackup/bin/admincmd/nbstlutil inactive  -backupid car-sql1_1353898826
Operation not successful: invalid request

 

no joy mark... try to inactivate the pending operation on a image but get the invalid request.. tried the command with other parameters such -force and along with -lifecycle policy name but no luck..

 

 

 

Bugs comes in through open windows!

Mark_Solutions's picture

I have just done this for 3500+ images (dont ask!!)

First i did the nbstlutil cancel -backup id **** -force

Then i ran bpexpdate -backupid *** -d 0 -nodelete -force

This has cleaned up all of my pending import images successfully.

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

SOLUTION
Spartacus81's picture

Dam.... all gone.. all back to Blue.. thanks Mark...

Bugs comes in through open windows!

Mark_Solutions's picture

GREAT!!!

And only took 3 months!!

So nice when something like this finally gets resolved - thanks for keeping us updated and your patience throughtout.

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.