Video Screencast Help

BE2014 DLM not delete already expired backup sets

Created: 12 Jun 2014 • Updated: 16 Jun 2014 | 44 comments
ksattler's picture

Hello,

I have upgraded a BE2012 SP4 installation to BE2014. The backups go to a DataDomain DD160 using as a OpenStorage device.

There a lot of backup sets which are already have been expired. The DLM of BE2014 seems not deleting already expired backup sets (e.g. from BE2012).

For new expired backup sets the deletion works.

How can I delete the >1000 expired backup sets?

Operating Systems:

Comments 44 CommentsJump to latest comment

WWSupport's picture

We are experiencing a similar issue. We have just installed BE 2014 in a test environment and cannot delete expired backup sets that were created by 2012.

ksattler's picture

I have opened a ticket two days ago. Symantec knows of this issue and they are working on it.

ikmit's picture

Any update on this?

This problem effectively stopped this weekend's run of full backups

DaveyMG's picture

Same issue here after upgrading from BE2012 to BE2014, expired B2D backup sets aren't being deleted.

BE2014 doesn't appear to have a UI option to delete a backup set so even though I can see there are expired sets without any dependent backup sets, I can't delete them manually.

I've also found expired B2D sets that have Duplicate-to-Tape backup sets dependent on them, even though there is a corresponding D2T set for the expired B2D set. I would expect D2T sets to only be dependent on other D2T sets.

I've got a case logged but so far I've been asked to install a hotfix related to a problem of backup sets being deleted prematurely. This didn't change the situation for me.

Frank_'s picture

I have exactly the same issue. Upgrade from 2012 to 2014 with the latest (cummulative HOTFIX 218257)

Expired media is filling up the disk because of dependency with duplicates on tape.

jerrys79's picture

ksattler,

do you have an update on the case you opened?

Thanks.

Celebi IT's picture

http://www.symantec.com/business/support/index?pag...

 Backup Exec button > Configuration and Settings > Backup Exec Settings > Storage > Allow Backup Exec to delete all expired backup sets

VirgilDobos's picture

Hi folks,

Starting with Backup Exec 2012, disk backup storage is managed by a new disk management function called Data Lifecycle Management (DLM). This allows BE to correctly manage disk targets and remove expired backup data. The Backup Definitions (jobs) have a setting for how long the backup data should be kept. After that timeframe has expired, DLM will delete the backup sets from Disk. DLM checks for any expired data every 4 hours by default. DLM is only used for disk targets (Backup to Disk (B2D) and Deduplication), while tape targets are managed by Media Sets like the previous versions of BE.

This being said, there are several considerations you should factor in. For example, if the backup image is the last recovery point, it will not be deleted until there is a newer one. The following article explains the two main situations where an expired backup sets will not be deleted after the expiration date and/or inability to overwrite or delete the last backup set: http://www.symantec.com/docs/TECH187957

I used several times the regkey mentioned within the article and that solved the problem: KEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Server\DeleteLastRecoverySetsOnceExpired. This registry key should delete all expired backup images.

Hope this helps.

Virgil Dobos | Sr. Technical Consultant | Condo-Protego | www.condoprotego.com 

Colin Weaver's picture

For BE 2014 this setting is exposed in the BE console so no need to edit the registry

Celebi IT already mentioned this earlier in the thread

DPILLC's picture

Same issue.  Upgraded to 2014 last week with all hot fixes.  Registry and config properly set as noted.  Still have backup sets from 2011 "marked" as expired with long ago expiration dates.

Moltron1's picture

I also upgraded from 2012 to 2014, and have OST / Data Domains.  There are quite a few expired sets, and they are not being removed.  Also, it seems like I can't expire them manually.  I've tried doing this, and get no error message, yet the set still shows the previous expiration.   Not sure whats going on with that.  I am expiring them so they'll get deleted and free me up some space?  Anyone also have this issue?

DaveyMG's picture

After 6 weeks my support case status is "Awaiting Hotfix". Not very meaningful - have they identified the problem and are working on a fix, or are they still trying to work out what the problem is?

Meanwhile my B2D storage is slowly filling up, even recently created backup sets that are set as expired are not being groomed nor will manually expiring them cause them to be deleted.

maurijo's picture

Call symantec, ask duty manager, raise to severity one, sit and wait.

kamel1204's picture

Hi

I have the same problem any news in this from Symantech Teck support

thanks 

Per Vaihinen Enter's picture

We have the same problem with upgraded Backup Exec 2012 -> 2014,  backup sets expired 2 months ago are still kept and slowly our DataDomain is filling up.

Have anyone got a resolution for this?

Should all having this problem open a case so Symantec would get more pressure to solve this?

DaveyMG's picture

I received an email from support today asking me to install SP1 to resolve the problem. I'll install next week.

DGar's picture

I have already installed Service Pack 1, and changed the DLM setting to allow BE to delete all expired datasets. And we still have the issue where the data will not delete. One of our BE drives has already filled and we only have 1TB available on the second drive. Has anyone found a fix to this issue yet?

dg

DaveyMG's picture

Same here. SP1 hasn't changed the situation.

I decided to try and delete one of my B2D volumes to remove the expired sets associated with it. I disabled the volume, selected to delete and also delete the backup sets from the catalog. After that I formatted the volume and used Configure Storage to add it back to BE. The volume added OK but the "Configuring Storage" part failed saying the volume was unavailable. Looking at the Storage screen showed the volume disabled. After enabling the volume and going through the add process again it configured OK except that all of the backup sets that were previously associated with the volume were still listed.

I repeated the above but gave the volume a different name but no change, the backup sets were still listed.

There seems to be something quite broken with the database. The only solution I see right now is to delete the whole database and face days of recataloging tapes.

Laurie_Downey's picture

After reaching out to some internal resources I have been advised that the issue discussed in this forum thread is possibly related to the following tech document: http://www.symantec.com/docs/TECH218049.

 

I say “possibly related” because detailed logs would need to be gathered for a precise determination on a per case basis. I would advise opening up a support case and referring to E-Track # 3527967 to confirm this information for your environment. 

 

Symantec has identified the issue as stated in the tech document above and we are looking into resolving the issue with the next service pack, planned for release in Q4, 2014.

 

 

Laurie Downey

Backup Exec

Advanced Tech Support

Symantec Corporation

www.symantec.com

Moltron1's picture

The workaround is to use tape?? We got rid of tape and spent lots of $$ getting 3 Data Domains and DD Boost!  First DD is now 100% full. 

Andy_S's picture

I have the same issue on two 3600 Appliances. SP1 hasn't been released for the appliances yet so no idea if that will fix things, but it doesn't sound too hopeful. I don't have the option of using tape and I have one DD that is now full, the other has more used space so I guess it'll only last a couple of more days. Are Symantec really suggesting we have to wait months for a fix? I would sooner clear everything down and start again but it sounds like DaveyMG has tried that without success.

jerrys79's picture

Is it safe to say that everyone that is having this problem upgraded from BE 2012?  Have any BE 2010 to 2014 upgrades experienced this? 

I wonder if what DaveyMG says is correct in that "The only solution I see right now is to delete the whole database and face days of recataloging tapes."

JonathanK's picture

I'm having the same issue on a clean install of BE 2014, no upgrade.  It only seems to occur when there is some kind of major issue (BE and/or OS crash, etc.).  I'm just using a simple B2D folder, so if space becomes an issue I just manually go delete the related files/folders on the B2D device, but the annoying expired backup sets remain in the details of the B2D device in the BE 2014 interface.

DaveyMG's picture

Following on from my previous comments, I persevered with trying to delete my B2D volumes and finally got the volumes reconfigured with all of the old expired sets gone. Something is still broken though as after I deleted my B2Disk1 volume I couldn't recreate it and had to use B2Disk1A as the volume name. If I tried to use B2Disk1 it would say that the name already exists. I'll need to wait a few days now to see if the DLM is deleting newly expired sets.

almost there's picture

Same issue here. Expired backup sets are being deleted. I saw Caleb's suggestion, but if you do this it will always delete expired backup sets and not leave the most recent if its the only copy. On top of this, the option to delete them manually is gone. Symantec???? Any suggestions?

Andy_S's picture

I've tried running the manual space reclamation as described in http://www.symantec.com/business/support/index?page=content&id=TECH192557 but it didn't have any effect.

Also trying to manually expire items by right clicking and selecting expire. Oddly at some point some did disappear from the list but I couldn't get any others to go and I have no idea why they were deleted.

Symantec tech support suggested it's a known issue and to prove it I should change a registry key and run SGMon for an hour or so and send them the logs:

HKey_Local_Machine\Software\Symantec\Backup Exec for Windows\Backup Exec\Server\CheckReclaimSpaceIntervalMinutes = 240 (Decimal)

Change to 30 (decimal) mins and restart the Backup exec services.

Did this and uploaded the files to them.

I then tried to again manually expire some already expired  datasets and they went. I managed to get rid of quite a few but while doing this I noticed the used dedupe disk space was increasing, not decreasing. After about twenty minutes it stopped allowing me to expire the sets.

I've rebooted the appliances but it still won't let me expire any more sets.

I've currently left the reg key at 30 minutes to see if that was what let me delete them but it appears to be back as it was and I can no longer delete anything and no idea why, for a short period of time, I could.

maurijo's picture

It is normal behaviour that your used dedupe space increases after deleting backup sets. When you delete a backup set your "delete action" goes in to the dedup queue. This produces transaction logs, which are sometimes quite large in size. When these transaction logs get processed by your dedupe the backup sets should be removed from disk and your used space should decrease. Please note that your space used will only decrease if the backup sets that you deleted actually contain unique data. If this is not the fact then you will receive no free space on your dedup.

Andy_S's picture

Has anyone had any success in deleting the expired sets. Both my appliances are now full and Symantec seemed to be pinning their hopes on SP1 fixing the issue although I noted that others had already applied SP1 without success. SP1 was released for the appliances yesterday, I have applied it to both devices and since then there has been no change.

ddominguez's picture

As Celeb IT said, you need to activated the Auto-Delete option 

Backup Exec button > Configuration and Settings > Backup Exec Settings > Storage > Allow Backup Exec to delete all expired backup sets.

When you do it, all expired backup sets will be automatically deleted (This could take some minutes). However, you will lost all expired backup sets, if you want to conserve some backups (may be if you are not sure if you have a recent backup) you can change the expiration date so you can conserve some backups.

After that you can deactivate that option again. Remember that DLM works saving all backup set until you have a recent full backup.

porka's picture

So I like others here have had this issue for sometime and just to reiterate service pack 1 makes no change to the issue nor does ticking the allow backup exec to delete all backup sets. I have 38 sites running backupexec 2014 and all are affected in the same way with at least 1 backup set that expired long ago and will not delete. The critical issue at one of the sites is that it now has a nas storage system for backups thats full because I have two backup sets totalling 3.8tb that expired according to backup exec on the 25th of august. Its ridiculous to suggest a workaround of backing up to tape, what about the people like me who dont have a tape drive whats the workaround then ???? Oh wait I think I know the workaround is stop using a ridiculous product and choose another vendor

DaveyMG's picture

I completely uninstalled BE2014 just over a week ago, deleted the database and reinstalled BE, installed SP1 and the hotfix, recreated backup jobs. So far I have some Exchange backup sets that show as expired on my disk storage and a couple of SQL server backups sets that show they are expired but show a dependent Tape backup set. Otherwise the rest of the backup to disk sets from other servers appear to be expiring and deleting OK. Cataloging of tapes from the previous installation has thrown up a different set of problems but I'll raise those elsewhere.

norm.tallant's picture

I'm expeirencing a really similar issue.  I have a fresh install of BE 2014 on Server 2012 R2.  It's a pretty simple setup wth only one server and one B2D device.

Due to a power outage, we had to shut down the server in the middle of backups on a weekend.

Now, even though I have the "Allow Backup Exec to delete all expired backup sets" box checked, and I've expired every backup on the server, and the files got deleted, BE2014 is reporting that it's using 5tb of space on my B2D volume.  I sure wish I could find out what's using that space!

Do I need to have a full backup + duplicate complete sucessfully for BE to release that disk space?

CinColo's picture

Make sure it's not trying to backup your external drive (i.e. the media you're using for backup).

CinColo's picture

In my case, this was a new install of 2104 on Server 2012 standard.  We back up to USB connected external hard drives, which are identical (which raised a red flag for the technician).  We were seeing that the backup sets for the new drive still showed the information for the previously connected external drive (which had not expired yet).  I believe we've found a solution but won't know for sure for a couple weeks until we have new data backed up to the drives.  The support person informed me that sometimes Backup Exec won't detect a new drive is attached if it has the same ID as the other drives, regardless of whether the data on the attached drive had expired.  I did notice this was odd when I first configured my backup job and attached each drive because I had only 1 name for the drives (Weekly Backup).  (The backups would run fine as long as I manually deleted data off them.).  In the past versions of BackupExec, I had to name each drive differently. It turns out you still need to name each one differently.  I've checked 2 of our drives so far, and they both had the same Unique ID. It looks promising now as Backup Exec forced me to name the second drive differently. You may already have unique names for your drives.  However, if they both have the same unique IDs, you can still be having this problem.  Below are the steps we followed:

  1. Confirm the ID for each external drive by following the steps listed on this link.https://www-secure.symantec.com/connect/blogs/changing-diskid  (also listed below).
  2. If the drives are all the same, modify the IDs so each drive is unique.
  3. If you're OK with starting clean, delete ALL the data on the external drive and go back into Configure Storage.  Delete the prior drive you had been using. Configure a new one-and this time it should prompt you to enter a unique name.

Steps for #1-Using the Windows PowerShell.

  1. Attach first disk to your server and type DISKPART to find the signature.
  2. Type LIST DISK.
  3. Type SELECT DISK #, where # is the disk number in you system (use Disk Management to find disk number if you are uncertain).  (i.e. "Select disk 2")
  4. Type UNIQUEID DISK.  This will give you a 4 byte disk ID e.g. "e9eb3aa5"
  5. Now attach the second (or third, etc) and do the same, find the UNIQUE ID, which may likely show the same ID as the first. With the same DISKPART tool you can overwrite the ID with a new ID, try incrementing the number with 1 or changing the last letter for each new disk.
  6. After you know all the unique IDs (which hopefully are the same), use the command UNIQUEID DISK ID=xxxxxxxx, where the x's is the new id you chose.
  7. After the ID is changed, delete all BE folders from that disk (including /BEControl and /BEdata), disconnect the USB drive, and reconnected it.
  8. This time BE Storage does not find the disk in the list and you can Add a new disk.

    http://www.symantec.com/connect/blogs/changing-diskid

UPDATE:  GIVING EACH DRIVE A UNIQUE ID FIXED THE PROBLEM FOR ME.

pkh's picture

BE recognises a disk by its DISKID and not by the name of the disk.  This is known for a long time.  See the blog.

ksattler's picture

Hi,

my case is open since Jun 2014. Round about every two weeks a got a feedback like this "Just keeping you updated that I am still awaiting further updates from engineering team on this incident and will contact you as soon as have further news on this issue."

I'm wondering that I have not received an information from the support that SP2 was released in December.

Does it really fix the issue?

mgmasterv2's picture

Hi,

any news on youre case? is youre problem solved?

no i updated my server to SP2 and nothing happend.

grumhelden's picture

Clean install on new servers, backing up a clean install on new hardware (BE2014, VMWARE 5.5), all patches applied *before* even establishing trust with servers.

And the disks get full and stay that way.

No upgrades, all Service Packs and hotfixes applied.

Product does not work.  Pure and simple.  It just doesnt work.

EDIT:  Just to contextualize, I am supposed to be taking my family on a day out, instead I'm chasing my tail trying to fix a broken product.  One that has no competition at the price point and which should at least have had this issue fixed by now.  So thanks for putting my family *and* my business at risk.

mgmasterv2's picture

as a person who recommends BE , im so sorry to hear that.

mgmasterv2's picture

i have same problem with  BE 2014 SP2 with local deduplication Disk updated from BE 2012 SP4 , i tried to expire backupset s manually & to recliam disk space manually (TECH130103) serveral time .backup sets disappear and after few minutes comes backup.

Strange thing is the only  backup sets that are not deleting are "Exchange 2013" VM & Database's backups!