Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

Netbackup 7 - Incorrect disk pool available space

Created: 26 Jul 2010 • Updated: 06 Sep 2010 | 12 comments
This issue has been solved. See solution.

Hi,

I'm currently testing Netbackup 7 in our lab. I created a media server dedup disk pool (on a Win2k8 R2 Std 64 bits server). Backups were working fine until recently. I'm now getting error 129 (insuffisent disk space). After looking at the drive, I only had 18gb left (the drive is 1TB). I went in the catalog and expired a bunch of older backups. My disk space is now back to 120gb but for some reason, Netbackup's disk pool properties still report only 18gb as available space. I've tried restarting services (and even rebooting the server) but nothing changed. I figured maybe it would get updated by one of the admin tasks that runs every night so I left it like that for the week-end but this morning, the console still shows 18gb available space. Is there a way to force Netbackup to recalculate the available space?

Also, on a related note... looking at the disk pool properties, there is a "Disk volumes" section where it reports my 18gb free space. I've search a lot but couldn't find a way to add a 2nd volume to the disk pool (didn't find anything in the admin guide or the console). Since the section has a "s" to volumes, I'm guessing it is doable. Can anyone point me in a direction in case I'd eventually need to add a 2nd lun to the server for additional space?

Thanks a lot for your help!
Philippe

Comments 12 CommentsJump to latest comment

AAlmroth's picture

You could try to do an inventory of the disk pool. This is available in the Disk Pools view in the Admin Console, right-click on the disk pool and select Inventory... Maybe it will work.

You could also run nbdevquery -listdp -stype PureDisk -U and see what it says. Also, replace -listdp with -listdv and you get the status of the volume.

AFAIK, no current OST device works with multiple volumes. For PDDE, you would rather have to extend the file system using either Windows disk manager or use Storage Foundation 5.1 and then create a concat device, and grow the file system.

/A

pmorin's picture

Hello,

Thanks for the quick reply! I had already tried the inventory but it only seem to recalculate usable/raw sizes. Doing the inventory didn't have any impact.

The listdp returned only global info about usable/raw sizes which are correct (1023gb). The listdv command returned the following:

Disk Pool Name      : LabDiskPool
Disk Type           : PureDisk
Disk Volume Name    : PureDiskVolume
Disk Media ID       : @aaaab
Total Capacity (GB) : 1024.00
Free Space (GB)     : 18.32
Use%                : 98
Status              : UP
Flag                : ReadOnWrite
Flag                : AdminUp
Flag                : InternalUp
Num Read Mounts     : 0
Num Write Mounts    : 1
Cur Read Streams    : 0
Cur Write Streams   : 0

As you can see, that command still reports 18gb of free space while Windows report 120gb available. If you have any other idea, let me know!

Thanks! :-)

Nicolai's picture

 I just wanted to share this command with you.  This command provide stat of the configured PDDE.

# /usr/openv/pdde/pdcr/bin/crcontrol --dsstat 1

************ Data Store statistics ************
Data storage       Size   Used  Avail Use%
             590.0G 463.9G 126.1G  79%
Number of containers             : 121
Average container size           : 247650044 bytes (236.18MB)
Space allocated for containers   : 29965655409 bytes (27.91GB)
Space used within containers     : 28524718837 bytes (26.57GB)
Space available within containers: 1440936572 bytes (1.34GB)
Space needs compaction           : 75511082 bytes (72.01MB)
Records marked for compaction    : 1049
Active records                   : 242823
Total records                    : 243872

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

AAlmroth's picture

In addition, you could try to manually run the PDDE maintenance processing;

install_path\Veritas\pdde\Crcontrol.exe --processqueue
install_path\Veritas\pdde\pdcr/bin/crcollect -v -m +1,+2 --noreport

And to watch the result, use the command Nicolai recommended.

Also, If you change the available space outside the control of NBU/PDDE, such as deleting unrelated files also stored on the volume, you have to restart NBU processes in order for it to discovered the new space.

/A

pmorin's picture

Hello,

Thanks again for the replies! Here is the output of crcontrol --dsstat 1:

************ Data Store statistics ************
Data storage       Size   Used  Avail Use%
                 1024.0G 669.0G 355.0G  66%
Number of containers             : 3590
Average container size           : 268435456 bytes (256.00MB)
Space allocated for containers   : 963683287040 bytes (897.50GB)
Space used within containers     : 712623327356 bytes (663.68GB)
Space available within containers: 251059959684 bytes (233.82GB)
Space needs compaction           : 974237482 bytes (929.11MB)
Records marked for compaction    : 11871
Active records                   : 7032121
Total records                    : 7043992

I'm not quite sure what this is supposed to show. My disk pool is indeed 1TB but has 120gb of free space while it is reporting 355gb here.

I also ran the other 2 commands. Crcontrol.exe --processqueue only outputs "OK" and returns to the prompt. I waited a couple minutes to be sure it would have time to process the queue and then ran crcollect -v -m +1,+2 --noreport which gave me this result:

modes         : +1,+2
report        : disabled
storagepool id: 39
start         : Tue Jul 27 07:22:16 2010
.. 10000 records processed
.. 20000 records processed
.. 30000 records processed
...
.. 7030000 records processed
.. 7032121 records processed
finish        : Tue Jul 27 07:23:50 2010
duration      : 0d:0h:1m:34s

statistics for mode 1 (SO without any reference to a DO):
processed            : 7031455
detected successfully: 0
detection error      : 0
executed successfully: 0
execution error      : 0
statistics for mode 2 (DO without with only task IDs):
processed            : 666
detected successfully: 50
detection error      : 0
executed successfully: 50
execution error      : 0

Running crcontrol --dsstat 1 again returned the exact same output as above.

Obviously, I didn't mess with the files on that drive. Anything that's been deleted was done through catalog expiration and has been done by Netbackup itself. Just to be sure though, I rebooted the whole server so that all services and w/e else could be resetted. Still reporting wrong available space on my disk pool :-(

If you have any other suggestions, let me know!

Thanks!

AAlmroth's picture

Coming to think about it, it could be the "built-in" 10% threshold in PDDE/PD. Thus, it always report 10% less available in the disk pool, than is available, and 1TB would be ~100GB, so your 18GB sounds right.

In PureDisk, this is possible to set using the webGUI, but I remember having seen it in a configuration file in the PDDE directory structure, but as I don't have a PDDE at hand, I cannot verify.

/A

pmorin's picture

If you're talking about the disk volume's high water mark, it is set to 98% right now. When I look at the disk pools section in the console, it says the pool is 98.17% full (which is exactly the free space it is reporting: 18.67gb / 1023.99gb). At this point, even catalog backups (which are just a couple hundred mb since this is just a lab environment) are failing because it thinks it doesn't have enough space.

I'll keep searching on my side. If I find something, I'll let you know.

Thanks!

AAlmroth's picture

No, it is not the NBU highwater mark, but an internal threshold in PDDE/PD. So in effect you have 12% you cannot use, so ~120GB in your case.
In PureDisk it is configurable in the GUI and is called Lowspace Threshold in the Content Router->Spooler section.

Look at another thread in the PureDisk forum called "PDDO diskpool space free discrepancy", and you will probably have the procedure how to increase the available space, but again, you should rather extend your storage to solve the problem...

/A

pmorin's picture

I've tried what was described in the forum post you mentioned but it didn't work. When I look at the backups in the catalog, there is only 3 backups left. All others expired (remember this is just a test environment and I hadn't done any new testing for the last 2 weeks which was my retention). Yet, the console is still only reporting 18gb. I've manually expired the last 3 backups and rebooted the whole server just to be sure all services would be restarted but still 18gb with 120gb left.

I'm going to try a backup and see if it overwrites the old expired data (which it should) and then maybe it will update its available space. Otherwise, is there a command that can be used to tell Netbackup to delete all expired data from disk?

AAlmroth's picture

Perhaps it is because PDDE still belives the images are referenced.

Try;
install_path\NetBackup\bin\admincmd\nbdelete -allvolumes -force
install_path\NetBackup\bin\admincmd\bpimage -cleanup -allclient 

Then wait for PDDE to run its maintenance tasks for cleanup, or run them manually;

install_path\Veritas\pdde\Crcontrol.exe --processqueue
install_path\Veritas\pdde\pdcr/bin/crcollect -v -m +1,+2 --noreport

/A

pmorin's picture

Finally found the issue! It is actually a bug in v7.0. I just found the hotfix:

BUG REPORT: Use% in nbdevquery is incorrect when using Media Server Deduplication disk pools
http://seer.entsupport.symantec.com/docs/350190.htm

The hotfix will also be part of v7.0.1

As soon as I installed the hotfix (be sure to close all services first or it won't be able to overwrite the exe and will still think the hotfix was properly installed!) and restarted the services, the available disk space went up to 894gb which seems right since I have 120gb free space left but all the files in the data folder have expired but haven't been deleted yet. I tried a backup and it worked!

Thanks again AAlmroth for the time you invested to help me out!

SOLUTION
djlaube's picture

Thanks for posting this information, this turned out to be my exact issue as well.

I've also posted it to our blog for reference and have included some of the useful commands from above on that post.
http://blog.lewan.com/2010/08/05/symantec-netbackup-7-incorrect-disk-pool-available-space-with-media-server-deduplication-puredisk-disk-pool/