Video Screencast Help
Search Video Help Close Back
to help
Not able to make it to Vision this year? Get a sampling in the Best of Vision on Demand group.

Netbackup 7.0 deduplication storage pool not releasing space

Created: 22 Jul 2010 | 14 comments
lpizzina's picture
0 0 Votes
Login to vote

Hello All, anyone using NBU7.0 with PDDE (deduplication storage media server) out there?

We're experiencing an annoying problem with the deduplication storage pool on our media server.
Once the dedup pool went beyond the high watermark (95%), of course all backup jobs going to it were failing with error 129.
Day after day, all images stored on it are expiring, one after the other, however the disk space is not released, it's still hanging at 96.3%, about 7TB

Initially we thought that until all segments of similar backups were alive, the space of the dedup pool would not be released. So, we expired manually most of the images in the dedup pool, no change.
Adding the size of all the images in the dedup pool, regardless of the deduplication benefits, add up to roughly 1.5TB. Considering deduplication, I'd expect that they should fit into a half TB or so. Instead, the used space still hangs around 7TB and doesn't move.

I opened a case with Symantec, but I'm still waiting.
If anyone had a similar experience, I'd appreciate your hints

Regs

Comments

bak's picture
22
Jul
2010
0 Votes 0
Login to vote

Symantec has documented bugs in PDDE deduplication at 7.0.  Your issue could be one that is fixed at 7.0.1.  It is still in the First Availability stage, but that code should become GA soon.

http://seer.entsupport.symantec.com/docs/350190.htm

http://seer.entsupport.symantec.com/docs/350196.htm

Marek Kedzierski's picture
22
Jul
2010
2 Votes +2
Login to vote
lpizzina's picture
25
Jul
2010
0 Votes 0
Login to vote

I forgot to mention that Etrack ET1984226 was already applied to master & media servers a month ago when this problem was seen. Apparently it fixed the problem, as dedup pool occupation instantly dropped below 80%.
However, in the meantime we added more clients to the protection domain and saw that the dedup pool occupation was raising again, little by little, but we thought it was correct as the data to backup was growing.
Then, after hitting the high watermark again, we're getting the same problem. No backups are going to the dedup pool, images contained there are expiring day after day, we calculated that the images contained in the dedup pool should total 1.5TB (without keeping in account deduplication...) but the dedup pool occupation is still 7TB and doesn't move.
There must be some other bug. Symantec is helping us with the investigation.

Regs

Evan's picture
05
Aug
2010
0 Votes 0
Login to vote

how did this end up? is it still an issue? I am getting this as well!

Thanks

Evan

Systems Integrator

Emmanuel College

Boston, MA

Tomer Gurantz's picture
11
Aug
2010
0 Votes 0
Login to vote

Also very interested in the outcome of this one.

Principal Learning Consultant with Symantec Education Course Development

Scott Chapman's picture
11
Aug
2010
0 Votes 0
Login to vote

I'd be interested to find out if this is a Windows or UNIX PDDE media server.

Thanks.

NetBackup 7 on Solaris 10.  Protecting Solaris 10 and Windows.

svcnbu's picture
01
Sep
2010
0 Votes 0
Login to vote

@ 7.0.1 having the same issue.

It's a test environment 7.0.1, I've expired all backup and run the following:

a. Force fragments and image cleanup:
nbdelete -allvolumes and bpimage -cleanup -allclients
b. Media Server Deduplication Pool cleanup:
crcontrol.exe --processqueue and crcollect -v -m +1,+2 --noreport

But still Disk space is not clear up.

emret's picture
18
Jan
2011
0 Votes 0
Login to vote

having the same problem

did anyone found any solution about this problem ?

abagutla's picture
21
Feb
2011
0 Votes 0
Login to vote

7.0.1 me too

I have the same problem. None of the expired images release space at disk pool.

No news?

Thanks.

Y_K's picture
21
Feb
2011
0 Votes 0
Login to vote

I have a similar problem. I

I have a similar problem. I preformed the following:

On the master:

1. bpimage -cleanup -allclients 

2. nbdelete -allvolumes 

then on the media server I ran the following commands:

3. crcollect -v -m +1,+2 

4. crcontrol --processqueue 

I repeated step 3 & 4 multiple times. I did see about 20% disk usage reduction but I was expecting more.

thanks

Marianne van den Berg's picture
21
Feb
2011
0 Votes 0
Login to vote

Has anyone actually logged a

Has anyone actually logged a Support call yet?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows.
Handy NBU links

Y_K's picture
21
Feb
2011
0 Votes 0
Login to vote

Yes... I have a call open,

Yes... I have a call open, they have not found any problems yet.

Y_K's picture
21
Feb
2011
0 Votes 0
Login to vote

Problem fixed

OK... I think my problem has been fixed. I had to run the following commands 3 times:

1.  crcollect -v -m +1,+2

Wait until it finished, it might take a little while.

2. crcontrol --processqueue 

this will take a long time to process everything in the queue that was processed by the previous command. To see if this process is still running, tun the following command crcontrol --processqueueinfo. you should receive no for busy and for pending.

I ran through this sequence 3 or 4 time (it took me 4 days) and not I am down to 3TB usage vs. 8TB when I started 4 days ago.

hope that help.

Thanks

Baka-kun's picture
29
Mar
2011
0 Votes 0
Login to vote

I also have this issue in my

I also have this issue in my test enviroment. The storage pool did not release space to file system. This is what I do.

1. Cause this is test environment so I expire all backup image on the storage pool. That will make all of my container file contain no data and can be remove.

2. I use step as Y_K mention above until there are no SO, DO records available and no queue to process. You can use command "Crcontrol.exe --queueinfo" to check if there are any queue to process.

3. I use command "Crcontrol.exe --dsstat 1" to check the available space that can be reclaim.

4. Run command "Crcontrol.exe --compactreleasesp 100 10 1". This will compact your container file. You can check status with command in step 3. You will see parameter "Space needs compaction" to be 0 MB

5. Run command "Crcontrol.exe --compactreleasesp 100". This will release space by delete empty container file 1 file/time you run command.

hope this help.