Netbackup/Deduplication Space Issues
I'm currently troubleshooting an issue regarding space on a 40 TB VLS, and am using NB 6.5.5 on a windows 2003 SP2 enterprise edition as the master. If more information is needed to provide relevant help later I will provide it.
For now lets jump in to the problem I am having. We have upwards of 75 clients writing full backups to this VLS, and we do a weekly, 6 month retention backup, and a daily 2 week retention backup of most of these clients. Last week we ran out of space entirely on the VLS, and now I am being asked to troubleshoot the issue. I have a bit of experience with VLS as well as a significant amount of experience managing backups on a day-to-day basis, and have been troubleshooting issues relating to this for approximately 2 years now. The first thing I did was check each client writing to this VLS to see if for some reason the images werent expiring from the catalog, as expected all of the clients were inside or right on their 6 month window, so I was fairly content there. I also checked frozen tapes and for any other of the more obvious issues. I spoke to a few senior staff who recommended I expire some older images in order to free up space. I had done this in the past for other clients and in semi-similar situations so from the catalog I went through each client that writes to this storage unit and expired a months worth of images (with some exeptions, I had not yet received approval for certain clients). After completing these steps and seeing the images expired from the catalog I ran the image cleanup command(Not sure this is necessary), logged on to the VLS, enabled writes, relabeled all available unassigned media relating to this storage unit in NETBACKUP, then reclaimed space on the VLS, as well I took a count of unassigned media assocated with this volume pool (232 unassigned tapes). Naturally I didn't expect this to immediately clear up the issues but by 15:00 the following day I noticed I went from .8 TB free on the storage unit to 3.11TB free. I wasnt' really happy with this so, and by this time I had the approvals to delete the rest of the images that I had not before the were older than 5 months. I went ahead and did this and waited yet another day, following the same steps before relabeling tapes and also reclaiming space, and monitoring the unassigned tapes in my scratch pool in netbackup (The number went up to 433 unassigned tapes).
In lieu of these issues happening I started reading some surrounding documentation relating to VLS / Netbackup and started comparing cartidges on the VLS and the images they hold, to what was in netbackup and noticed some discrepancies tonight. After going through about 30~ cartidges on the storage unit I am seeing images about 13-16 months old that should have already been expired stored on cartridges in the storage unit. I found about 15 instances of this on the 30 cartridges I checked and im wondering why it is that although I have removed these images from netbackup, why the images are not removed from the storage unit, or at least netbackup is not re-using this space. I know in the past NB would not always delete images but would for sure see the space as usable. Is it because these tapes are marked as full?
If necessary I can provide signifcantly more documentation surrounding my issue and apologize if anything I stated was unclear. As it stands right now I am about to drop retention down from 5 months to to 4 months.