Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

How do I delete backups on an old DiskPool

Created: 05 Jul 2014 • Updated: 08 Jul 2014 | 24 comments
This issue has been solved. See solution.

Hello everyone,

I hope you can help me out.  Several months ago, my company added a new MSA disk pool to replace our aging DataDomain disk pool.  All backup policies were reconfigured to stop using DataDomain and, originally, the usage on DataDomain dropped as backups aged and expired.  The problem is that the Data Domain usage only dropped to 14% and has been stuck there for over a month.  When I go into the NetBackup GUI (version 7.1) and navigate to "Media and Device Management" > Devices > Disk Pools, it displays 2.5677 TB used oiut of 18.1005 TB total (about 14%). 

I'm not sure why these backups are not expiring.  After doing some reading, I ran the following from D:\Program Files\Veritas\NetBackup\bin\admincmd on my NetBackup admin host:

bpimmedia -L -stype DiskPool -dp dd670
Backup-ID             Policy     ScheduleType
Copy  Frag  Expires     DiskType  DiskPoolName  DiskVolume  StorageServerName
-----------------------------------------------------------------------------
APP-SRV29_1357349450  APP_SRV29  FULL
1     1     1360027850  DiskPool  dd670         DDSTU1      inf-srv17
APP-SRV29_1357349451  APP_SRV29  FULL
1     1     1360027851  DiskPool  dd670         DDSTU1      inf-srv17
... (and on for hundreds of entries)

The server APP_SRV29 no longer exists so we certainly don't need to keep these backups (because all backups are for disaster recovery, not historical snapshots).  After some more reading, I found documents talking about bpexpdate.  Unfortunatetly, using this command gives me "no entry was found":

bpexpdate  -backupid APP-SRV29_1357349450 -d 0
Are you SURE you want to delete APP-SRV29_1357349450 y/n (n)? y
no entity was found

Oddly, even though the bpimmedia command shows the Policy of APP_SRV29, the bpimagelist shows nothing:

bpimagelist -policy APP_SRV29
no entity was found

I am very frustrated.  I have 2.6 TB of backups sitting in my disk pool that I can see but not clean up.  Can anyone offer me help on how to delete these backup IDs?  Any help would be greatly appreciated.

Ken

Operating Systems:

Comments 24 CommentsJump to latest comment

Riaan.Badenhorst's picture

Hi,

 

The following will wipe the entire DD.

bpexpdate -stype DataDomain [-dp disk_pool_name] [-nodelete]

nbdelete -allvolumes

 

Regards,

Riaan Badenhorst

You need an OpenVision to see the truth about Backups. Restores are a plus. But that's just Semantics ;)

ITs easy :)

khemmerl's picture

Thanks for the reply Riaan.Badenhorst.  I think I'll use that as a last resort.  I was hoping to be able to go through and clean up backups that are not needed anymore and then review the remaining to ensure there are backups to the new disk THEN use the command you've provided. 

Does Netbackup somehow treat diskpools so differently that there's no way to delete individual backups?

Ken

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

Marianne's picture

I find it strange that bpimmedia is still listing this image: APP-SRV29_1357349450 (and the other one too), as the expiration date (1360027850) was more than a year ago:

UNIX time 1360027850 is 02/05/2013 1:30am GMT

Try this:
nbdelete -allvolumes -force

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

khemmerl's picture

That command doesn't appear to do anything.

D:\Program Files\Veritas\NetBackup\bin\admincmd>nbdelete -allvolumes -force

D:\Program Files\Veritas\NetBackup\bin\admincmd>

The output of bpimmedia does not appear to be any different after running the nbdelete command

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

Mark_Solutions's picture

Did you try bpexpdate -backupid <Backup ID> -d 0 -nodelete?

Was it part of a SLP whose Lifecycle has not completed? If you you may need to cancel its lifecycle first.

There is also this one if you are really struggling:

nbstlutil remove_exp -force

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Marianne's picture

Also see what this command tells us:

bpimagelist - client APP_SRV29 -d 01/01/2013 -U

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mph999's picture

nbdelete -allvolumes -force will only 'work' if there is reference to the images in NBDB (in the correct tables) 

Else, the files on the STU are just that, files that NBU will ignore.

So. :

1.  Output of command from marianne above.

2.  Answer to Marks question as to if these are SLP images or not.

(I suspect not else when you try to bpexpdate them, it will actually tell you they are under SLP control, however, it does no harm to x2 check to avaoid any misunderstanding).

If 1/ and 2/ are 'clear' (ie. 1 shows nothing and answer to 2 is no) I suspect that in the past a bpexpdat was run with the nodelete option.  This clears the entries from the DB, but leaves the actual images on disk.  They then can't be deleted as NBU knows nothing about them - however, i could be wrong ...

No matter the way to investigate is the same:

Create an empty folder eg /data_unload

Run 

/usr/openv/db/bin/nbdb_unload /data_unload

This will create a  copy of NBDB, search every file for :

APP-SRV29_1357349450

grep 1357349450 *.dat

NOTE: Important to only search for the ctime from the backupid

May need to make other serchs using other backupids, but this is one you report the error on, so we'll try this first.

If entries apprear in some of the dat files, post up which .dat files they are and also a complete copy or the reload sql file which will also have been created.

Then, we identify which tables the .dat files reference, and knowing this can decide on the next step.

Hopefully, the backupids won't appear in any .dat files, in which case the files on the storage can simply be deleted via 'os' commands, however, we would need to double check the understanding of the issue before doing this, as it's fairly terminal ...

If they do appear in some .dat files, then depending on which ones, depend on what we do.  It could also mean other 'seraches' being required, not sure yet.

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
khemmerl's picture

Your first command give "no entity was found":

bpexpdate -backupid APP-SRV29_1357349450 -d 0 -nodelete
Are you SURE you want to delete APP-SRV29_1357349450 y/n (n)? y
no entity was found

The second command gives "Not implemented yet"

nbstlutil remove_exp -force
This operation will remove all expired image fragment entries in the database.  Any files containing backup images corresponding to these en
.  Do you wish to proceed? (y/n) >y
Not implemented yet.

nbstlutil does manage to see the backups though...

nbstlutil list
V7.0.1 I inf-srv17 APP-SRV29_1357349450 APP-SRV29 1357349450 APP_SRV29 13 0 Disk-Disk-Tape-Monthly 3 1357349941 *NULL* 5  00000000-0000-0000884
V7.0.1 C inf-srv17 APP-SRV29_1357349450 1 1360027850 1360027850 dd670-1dsu 3 0 0 0 0 *NONE* 1360027850 0
V7.0.1 F inf-srv17 APP-SRV29_1357349450 1 1 0 @aaaac inf-srv17 *NULL* 0 6 1 5560740864 1 @aaaac
V7.0.1 C inf-srv17 APP-SRV29_1357349450 2 1360027850 1360027850 dd670-2dsu 3 632768 0 0 1 *NULL* 1360027850 1357350012
V7.0.1 F inf-srv17 APP-SRV29_1357349450 2 1 2 @aaaae inf-srv17 *NULL* 0 6 1 5560740864 1 @aaaae
V7.0.1 I inf-srv17 APP-SRV29_1357349451 APP-SRV29 1357349451 APP_SRV29 13 0 Disk-Disk-Tape-Monthly 3 1357350864 *NULL* 5  00000000-0000-0000683
V7.0.1 C inf-srv17 APP-SRV29_1357349451 1 1360027851 1360027851 dd670-1dsu 3 0 0 0 0 *NONE* 1360027851 0
V7.0.1 F inf-srv17 APP-SRV29_1357349451 1 1 0 @aaaac inf-srv17 *NULL* 0 6 1 6437824512 1 @aaaac
V7.0.1 C inf-srv17 APP-SRV29_1357349451 2 1360027851 1360027851 dd670-2dsu 3 632793 0 0 1 *NULL* 1360027851 1357350962
V7.0.1 F inf-srv17 APP-SRV29_1357349451 2 1 1 @aaaae inf-srv17 *NULL* 0 6 1 6437824512 1 @aaaae
317

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

khemmerl's picture

In reply to the suggestion from Marianne, I tried the bpimagelist specifying both APP_SRV29 and APP-SRV29 (dash instead of underscore).  Neither command returns anything:

bpimagelist -client APP_SRV29 -d 01/01/2013 -U
no entity was found

bpimagelist -client APP-SRV29 -d 01/01/2013 -U
no entity was found

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

khemmerl's picture

Regarding the question about whether these files are SLP images or not:  I'm sorry but can you explain how to answer that question?  

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

khemmerl's picture

To answer the SLP question:  I have a Storage Lifecycle Policy named "Disk-Disk-Tape-Monthly" which is supposed to

  • send a backup to the local DataDomain for a one month retention
  • replicate to the remote DataDomain for a one month retention
  • replicate to tape for offsite storage for one year

When I run "nbstlutil list", I see:

V7.0.1 I inf-srv17 APP-SRV29_1357349450 APP-SRV29 1357349450 APP_SRV29 13 0 Disk-Disk-Tape-Monthly 3 1357349941 *NULL* 5  00000000-0000-0000884

so it appears that these are SLP backups.

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

khemmerl's picture

> If entries apprear in some of the dat files,
> post up which .dat files they are and also a
> complete copy or the reload sql file which
> will also have been created.

"grep 1357349450 *.dat" returned one entry in 808.dat and two entries in each of 810.dat and 811.dat.

grep 1357349450 *.dat

808.dat:'1000003','APP-SRV29_1357349450','APP-SRV29','1357349450','A
PP_SRV29','13','0','Disk-Disk-Tape-Monthly','3','1357349941','','5',
'',0x00000000,'0','0','2013-01-05 01:38:04.949000','2013-01-05 01:49
:13.507000'

810.dat:'1000003','APP-SRV29_1357349450','1','0','1360027850','13600
27850','dd670-1dsu','3','0','0','0','0','0','*NONE*','0','0','136002
7850','0','2013-01-05 01:38:04.949000','2013-01-05 01:49:13.351000'

810.dat:'1000003','APP-SRV29_1357349450','2','1','1360027850','13600
27850','dd670-2dsu','3','632768','0','0','1','1357350012','','1','0'
,'1360027850','0','2013-01-05 01:39:01.842000','2013-01-05 01:49:13.
366000'

811.dat:'1000003','APP-SRV29_1357349450','1','1','0','@aaaac','10000
03','0','0','6','1','5560740864','1','@aaaac','','2013-01-05 01:38:0
4.964000','2013-01-05 01:38:04.964000'

811.dat:'1000003','APP-SRV29_1357349450','2','1','2','@aaaae','10000
03','0','0','6','1','5560740864','1','@aaaae','','2013-01-05 01:40:1
2.651000','2013-01-05 01:40:12.651000'

"reload.sql" attached as requested.

AttachmentSize
reload.sql_.txt 526.96 KB

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

Marianne's picture

I suggest you log a Support call right away.

You will need assistance from NBU support engineer to fix inconcistencies.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

mph999's picture

OK, so if you run :

nbstlutil stlilist

This will give a slightly different output :

Example

V7.6.0 I womble_1404298590 slp1 3

If there is a 3 at the end, the image is complete and is not under SLP control

If it is a 1, or a 2 the image is either not started (1) or in progress (2) which will be stopping them being deleted.

Could you also try :

bpimagelist -backupid APP-SRV29_1357349450 -l

This should in theory return no entitiy found (based on your other bpimagelist command) but will be interesting to see for sure.  However, on the flip side, seeing nbstlutil list sees the image, so should bpimagelist ...

So in summary.

nbstlutil list shows the imeage

bpimage list doesn't (but please check using my command above backupid)

Even if under SLP (ie. 1 or 2 in nbstlutil stlilist output) this should't cause the no entity found, it should still be found, it would just fail to delete.

Other thing to check would be bpimagelist -l -d 01/04/13 -e 01/06/13 output, do you see the backupid in there (1357349450 converts to Jan 05 13

... in other words, is it list when not specifiacally searched for by backupid.

Got to pop out, will check back later

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

OK, scrap my above post - sorry didn;t see your post about the .dat file.

Let me take a look at that before we do anything else ...

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

Need to pop out so will do when I get back, in about an hour ...

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

Let me look before you call support (Mariaane is probablt right in that a support call will be needed) but let me chack as even with the delay of me going out, I will be quicker than support (they'll probably want to run some commands etc ... which prob aren't needed).

I'll forward the details to my ipad and look from there, try and shorten the time before I reply, otherwisew it'll be about an hour ...

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

OK.

808/ 810 /811 are Image / Copy / Frag tables

From reload file

INPUT INTO "EMM_MAIN"."EMM_Image" 
    FROM 'D:/PROGRA~1/Veritas/NetBackup/bin/admincmd/data_unload/808.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("MasterServerKey","BackupID","ClientName","BackupTime","PolicyName","ClientType","ScheduleType","StorageServiceName","StorageServiceState","TimeInProcess","ClassificationID","StorageServiceVersionNumber","OriginMasterServer","OriginMasterServerID","RequiredExpirationDate","ImportFromReplicaTime","CreatedDateTime","LastModifiedDateTime")
    ENCODING 'UTF-8'
go

call sa_unload_display_table_status( 17737, 80, 91, 'EMM_MAIN', 'EMM_MS_StorageServer_Connection' )
go

INPUT INTO "EMM_MAIN"."EMM_MS_StorageServer_Connection" 
    FROM 'D:/PROGRA~1/Veritas/NetBackup/bin/admincmd/data_unload/809.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("MediaServerKey","StorageServerKey","StateFlags","MaintOpPriority")
    ENCODING 'UTF-8'
go

call sa_unload_display_table_status( 17737, 81, 91, 'EMM_MAIN', 'EMM_ImageCopy' )
go

INPUT INTO "EMM_MAIN"."EMM_ImageCopy" 
    FROM 'D:/PROGRA~1/Veritas/NetBackup/bin/admincmd/data_unload/810.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("MasterServerKey","BackupID","CopyNumber","CopyType","ExpireTime","TryToKeepTime","Residence","CopyState","JobID","ExpirationType","MpxState","RetryCount","LastRetryTime","LifecycleDestinationTag","LifecycleSourceTag","DeferredDuplicationTime","ExpireLifecycleTime","IsReplica","CreatedDateTime","LastModifiedDateTime")
    ENCODING 'UTF-8'
go

call sa_unload_display_table_status( 17737, 82, 91, 'EMM_MAIN', 'EMM_ImageFragment' )
go

INPUT INTO "EMM_MAIN"."EMM_ImageFragment" 
    FROM 'D:/PROGRA~1/Veritas/NetBackup/bin/admincmd/data_unload/811.dat'
    FORMAT TEXT
    ESCAPE CHARACTER '\\'
    BY ORDER("MasterServerKey","BackupID","CopyNumber","FragmentNumber","ResumeCount","MediaID","MediaServerKey","StorageServerKey","StorageUnitType","StuSubType","FragmentState","FragmentSize","DeleteHeader","FragmentID","MediaExtents","CreatedDateTime","LastModifiedDateTime")
    ENCODING 'UTF-8'

So if bpimagelist is not showing them, clearly something is wrong - so I think a support call is a good idea as I suspect this will need investigation via webex and form what I see at the moment is going to need engineering.  I'll look later to see if I can see what is wrong with the entries, but that's not necessarily obvious unfortunately.

Log call, pls. post up case number and show them this post:
Also run
bpimagelist -backupid <backupid> -l 

If this command returns 'noentity found' then mention this also

bpimagelist -backupid <backupid> shows no entity found, but image is referenced in Image / Copy/ Frag tables of NBDB (808 /810/ 811.dat) (mention if it shows the output or not, presume not for the min))

And

bpimagelist -client APP_SRV29 -d 01/01/2013 -U
no entity was found

Run 

bpimagelist -l -d 01/04/13 -e 01/06/13 

Check if this shows the backupid or not

Include nbdb_unload output in the case, and nbsu -c -t

I'll check back later

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
khemmerl's picture

> nbstlutil stlilist
>
> If there is a 3 at the end, the image is complete and is not under SLP control

Results of "nbstlutil stlilist":

V7.0.1 I APP-SRV29_1357349450 Disk-Disk-Tape-Monthly 3
V7.0.1 I APP-SRV29_1357349451 Disk-Disk-Tape-Monthly 3

After just a quick check it appears that all rows end with a '3'.

> bpimagelist -backupid APP-SRV29_1357349450 -l
> should in theory return no entitiy found...

Correct:

bpimagelist -backupid APP-SRV29_1357349450 -l
no entity was found

> Other thing to check would be bpimagelist -l -d 01/04/13 -e 01/06/13

This returns some rows but not any for APP-SRV29.

bpimagelist -l -d 01/04/13 -e 01/06/13

IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434038 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434038 6640 1420506038 0 0 30119133 539 1 1 0 INF-SRV
42_1 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633683 2 0 6
5251 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NULL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 30119133 0 2 6 2 APA290 inf-srv17 65536 4713499 1357413731
11 0 *NULL* 1420506038 0 65546 0 0 0 1 1420506038 1357441439 0 *NULL
* *N
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434037 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434037 4639 1420506037 0 0 13360959 55285 1 1 0 INF-S
RV42L* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633682 2 0
 7246877 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *N
ULL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 13360959 0 2 6 5 APA071 inf-srv17 65536 430877 1357350254 9
 0 *NULL* 1420506037 0 65546 0 0 0 1 1420506037 1357439355 0 *NULL*
*NUL
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434036 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434036 2996 1420506036 0 0 6757899 35919 1 1 0 INF-SR
V42_* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633681 2 0
4714550 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NU
LL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 6757899 0 2 6 4 APA071 inf-srv17 65536 325282 1357350254 10
 0 *NULL* 1420506036 0 65546 0 0 0 1 1420506036 1357437398 0 *NULL*
*NUL
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434035 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434035 6301 1420506035 0 0 14874879 7923 1 1 0 INF-SR
V42_* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633680 2 0
992471 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NUL
L*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 14874879 0 2 6 6 APA071 inf-srv17 65536 639644 1357350254 0
 0 *NULL* 1420506035 0 65546 0 0 0 1 1420506035 1357440916 0 *NULL*
*NUL

I've checked the output of the "bpimmedia -L -stype DiskPool -dp dd670" command and there is no reference to INF-SRV42 in that output so these entries appear to be something different.

 

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

mph999's picture

OK, so it seems fairly clear that although there are entries in the image/ copy/ frag tables, bpimagelist is not reporting them - log call time I think ( please post case number up here )

Call details to include:

 

Issue:

Several months ago, my company added a new MSA disk pool to replace our aging DataDomain disk pool.  All backup policies were reconfigured to stop using DataDomain and, originally, the usage on DataDomain dropped as backups aged and expired.  The problem is that the Data Domain usage only dropped to 14% and has been stuck there for over a month.  When I go into the NetBackup GUI (version 7.1) and navigate to "Media and Device Management" > Devices > Disk Pools, it displays 2.5677 TB used oiut of 18.1005 TB total (about 14%). 

Investigations and research so far:

(For explanation purposes, backupid APP-SRV29_1357349450 is used, though many others are affected)

bpimagelist -backupid <backupid> shows no entity found, but image is referenced in Image / Copy/ Frag tables of NBDB (808 /810/ 811.dat)  This was determined by nbdb_unload on NBDB and then searching .dat files for the ctime.

And

bpimagelist -client APP_SRV29 -d 01/01/2013 -U
no entity was found

 

 bpimagelist -l -d 01/04/13 -e 01/06/13

This returns some rows but not any for APP-SRV29.

bpimagelist -l -d 01/04/13 -e 01/06/13

IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434038 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434038 6640 1420506038 0 0 30119133 539 1 1 0 INF-SRV
42_1 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633683 2 0 6
5251 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NULL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 30119133 0 2 6 2 APA290 inf-srv17 65536 4713499 1357413731
11 0 *NULL* 1420506038 0 65546 0 0 0 1 1420506038 1357441439 0 *NULL
* *N
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434037 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434037 4639 1420506037 0 0 13360959 55285 1 1 0 INF-S
RV42L* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633682 2 0
 7246877 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *N
ULL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 13360959 0 2 6 5 APA071 inf-srv17 65536 430877 1357350254 9
 0 *NULL* 1420506037 0 65546 0 0 0 1 1420506037 1357439355 0 *NULL*
*NUL
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434036 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434036 2996 1420506036 0 0 6757899 35919 1 1 0 INF-SR
V42_* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633681 2 0
4714550 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NU
LL*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 6757899 0 2 6 4 APA071 inf-srv17 65536 325282 1357350254 10
 0 *NULL* 1420506036 0 65546 0 0 0 1 1420506036 1357437398 0 *NULL*
*NUL
IMAGE INF-SRV42 0 0 8 INF-SRV42_1357434035 INF-SRV42 13 *NULL* root
Yearly 0 3 1357434035 6301 1420506035 0 0 14874879 7923 1 1 0 INF-SR
V42_* 0 3 0 0 0 *NULL* 0 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 633680 2 0
992471 0 0 *NULL* Disk-Disk-Tape-Yearly 3 1357434002 0 0 *NULL* *NUL
L*
HISTO -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
FRAG 3 1 14874879 0 2 6 6 APA071 inf-srv17 65536 639644 1357350254 0
 0 *NULL* 1420506035 0 65546 0 0 0 1 1420506035 1357440916 0 *NULL*
*NUL

 

bpimmedia does however show the images, but it cannot be expired with bpexpdate:

bpimmedia -L -stype DiskPool -dp dd670
Backup-ID             Policy     ScheduleType
Copy  Frag  Expires     DiskType  DiskPoolName  DiskVolume  StorageServerName
-----------------------------------------------------------------------------
APP-SRV29_1357349450  APP_SRV29  FULL
1     1     1360027850  DiskPool  dd670         DDSTU1      inf-srv17
APP-SRV29_1357349451  APP_SRV29  FULL
1     1     1360027851  DiskPool  dd670         DDSTU1      inf-srv17
... (and on for hundreds of entries)

 

The image is also found in nbstlutil stlilist output

"nbstlutil stlilist":

V7.0.1 I APP-SRV29_1357349450 Disk-Disk-Tape-Monthly 3
V7.0.1 I APP-SRV29_1357349451 Disk-Disk-Tape-Monthly 3

Though as the status is 3, it is no longer under SLP control, this of course should not affect the output of bpimagelist.

The server APP_SRV29 no longer exists so we certainly don't need to keep these backups (because all backups are for disaster recovery, not historical snapshots).  After some more reading, I found documents talking about bpexpdate.  Unfortunatetly, using this command gives me "no entry was found":

bpexpdate  -backupid APP-SRV29_1357349450 -d 0
Are you SURE you want to delete APP-SRV29_1357349450 y/n (n)? y
no entity was found

 

Please assist in determining why bpexpdate and bpimagelist don't find images that are contained in the image/ copy /frag tables.

After logging the call with the above details, please upload (or get ready to upload) :

nbdb_unload of DB (all files)

nbsu -c -t

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
mph999's picture

Just had a thought :

If you can, stop NBU and copy all the files in /usr/openv/db/data and upload these.

Upload these files also (this is a 'hot' backup of the NBDB, same as what the catalog backup does.

The idea for this is it allows your DB to be started up on a test server - you have to add an alias of your histname to /etc/hosts, or pref rename the test server to have the same servername as yours, then edit the following files to have the same hostname:

/usr/openv/netbackup/bp.conf
/usr/openv/var/global/server.conf
/usr/openv/db/data/vxdbms.conf
/usr/openv/db/bin/servername

This was support can investigate what's going on by running commands directly without having to bother you via email/ webex - hopefully ...

 

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
khemmerl's picture

Thank you so much for all your help.  What I never mentioned was that in addition to moving away from DataDomain, my company is also in the process of moving away from NetBackup.  I thought we had support to the end of this month but just learned that I was mistaken and that our support contract with NetBackup is already expired.  This means I will not be able to open a support ticket for this problem. 

As there are off-site tapes that won't expire for another two years, I'm looking at leaving NetBackup server running but doing nothing until the last of the tapes expires.  The DataDomain device has a sanitize function that securely wipes the disks that I'll use before shutting down the two units, removing them from my primary and disaster recovery sites and recycling.  I would have preferred to have things perfectly neat and orderly but will have to brute-force it a bit instead. 

Lesson learned:  1) Start cleanup long before your license expires.  2) Make sure you know when your support actually expires.

Thanks again for all your help.  Sorry I'm not able to open a ticket to find the resolution - I would have liked to get to the bottom of this.

Ken

Ken Hemmerling
Alberta Pensions Services Corporation
Database Administrator
5103 Windermere Blvd. SW
Edmonton, AB T6W 0S9

mph999's picture

Sorry to hear you are leaving NBU.

You could just delete the images from the disk directly.  I don't think this will cause any more issues - even on a fully working system, NBU would just delete the images from the catalog even if they aren't on the disk, it might throw an error somewhere but I think that would about be the worst of it.  Seeing you can;t delete the images, it'll just stay in that state, so you might as well reclaim some disk space manually.

Unfortunatley, unless one of the others comes up with some bright ideas I'm a bit stuck - this has got a bit 'complex' for a forum issue, needs to be a bit more hands on to make progress I think.

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
SOLUTION
Mark_Solutions's picture

Just realised that you are still on NetBackup 7.1 .. so that means there is one last possible solution here if you just want to get rid of everything from that disk pool.

First run:

nbdevquery -listdv -stype PureDisk

This will show all of your disk pools and their mediaid's (like @aaaae)

Once you have the mediaid for that pool use the following - using your mediaid in the command - do be sure of this though as it will get rid of everything in that disk pool!!!!!:

nbstlutil remove_all -mediaid @aaaae –force

See if that sorts it for you .. they dropped this command after 7.1 but if you are still on that version you can give it a try.

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.