Video Screencast Help

Cannot reclaim all space on deduplcation DATA in the MSDP Disk Pool

Created: 06 Jan 2013 | 12 comments
Leo46's picture

Hi Everyone,

 

Approciate and thoughts or Input I get. We running backups at Production to disk and replicate DATA to our DR site. We are running NETBACKUP 7.5.1 with Deduplcation enabled.

The issue that I am having is that the Disk space at DR which is 7.8TB is filling up much quicker than usual.

I have been running Garbage collections (Crccollect -v -m +1,+2) to speed up the Image clean-up process which runs for a few hours and clears up space. This works perfectly at Production however at DR only recent Images expired from Netbackup are cleared. I did some digging and noticed that in the  MSDP\Storage\DATA folder there are DATA segments ranging from a couple of months back which have already been deleted from the Catalog. Even if I do a catalog search for these images in the Netbackup GUI for these date ranges they are not visible however are still taking up 2.8TB of Space! The Garbage collection doesnt seem to clear any of this derefrenced\ Orpahend segments.

 

We have a policy which states we are not aloud to run replciation during business hours 8Am - 5PM, so everyday a few replication jobs are cancelled soim not sure if this is maybe causing this.

 

Please help, we have replcation jobs failing as we are running out of disk space.

 

Thanks in advance.

Comments 12 CommentsJump to latest comment

Andrew Madsen's picture

What does your SLP look like for retention at your DR site? The segments (I assume you are talking about the .bin and .bhd files) do not mean that there is anything in them. They are containers and to the file system they appear one size and to NetBackup they are another.

What is the output of /usr/openv/pdde/pdcr/bin/crcontrol --dsstat?

Last but not least if the target site of an AIR is not doing anything but receiving AIR then the deduplication will be the same as the source site. so if you have 24 TB at the source site you will need 24 TB at the secondary site.

The above comments are not to be construed as an official stance of the company I work for; hell half the time they are not even an official stance for me.

Nagalla's picture

hey,

did you Process the transaction queue.

You can also remove garbage data at any time. If you remove garbage data, you must also process the transaction queue twice Any time that you do something manually that generates transactions, you should process the transaction queue twice to complete the operation. The crcollect and crcontrolcommands are required for these manual operations. See Remove garbage data

Process the transaction queue

 
During the backup process, transactions are stored in a transaction queue. The transactions are located in a queue directory under the deduplication storage path. For the PureDisk server, the storage path is/Storage; for the MSDP server the path is customized during the installation. The queue directory contains files with the extensions .tlog and .delayed.
The backup jobs, queue processing, and many other operations can generate new entries in the queue. Therefore, when you process the queue, you will notice that some of the old .tlog files disappear, but new .tlog files are generated.
After one queue processing is finished, the oldest .tlog file should not be older than the start time of that last queue processing. Otherwise, contact support and provide the storaged.log.
To process the transaction queue from the command line on the MSDP media server or PureDisk Storage Pool server, perform the following steps:
  1. Check if another queue processing is running:
    crcontrol --processqueueinfo
  2. Repeat step 1 until pending and busy status show "no" as the result.
    Remark: one queue processing can take between a few minutes up to about a day. Progress can be seen in the storaged.log.
  3. Start the CR queue processing:
    crcontrol --processqueue
  4. Check if the queue processing is finished by using the following command:
    crcontrol --processqueueinfo
  5. Repeat step 4 until pending and busy status show "no" as the result.

The storaged.log is located in the following directories:

  • On a PureDisk Storage Pool server - /Storage/log/spoold/storaged.log
  • On a UNIX or Linux MSDP Media Server - <storage path>/log/spoold/storaged.log
  • On a Windows MSDP media server - <storage path>\log\spoold\storaged.log

 

see the below T/N for referrence.

http://www.symantec.com/business/support/index?pag...

hope this helps.

Marianne's picture

 

Seems this is a bug that is resolved in 7.5.0.4.

See this TN: http://www.symantec.com/docs/TECH190714

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Rickard Nell's picture

Ive experienced this in my environment as well, and even though we did install the 7.5.0.4 we still get this from time to time.

What I realised is that there are images/data on disk the Netbackup is not aware of therefore cannot mark the data/containers for expiration

This is what I do once a month:

Phase 1 import the pure disk server and wait for it to complete

I then view what images is on disk that is not in Netbackup's catalog.

I import all images prior to the retention period and wait for that to complete.

There after I expire those images that is before the retention period and run the garbage collection after the image cleanup job has completed.

 

 

Mark_Solutions's picture

I would like to see a crcontrol --dsstat if possible

As well as garbage collection and queue processing comes Compaction

Some of these only run one a month by default which with short retention periods may not be enough and you may need to set them to run more often to suit your needs and keep things clean.

It is is orphaned fragments even after 7.5.0.4 then it needs logging to ensure Symantec are aware and get the issue addressed - i have a feeling that there is a command that can be run to clean these up but cannot think of it right now - will get back to you if my grey matter starts working later!)

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Leo46's picture

Hi All,

I agree, this may be a Bug, I will update this week still and let the forum know on the outcome. attached is the Crcontrol --dsstat if interested.

Crcontrol --dsstat output.PNG
Andrew Madsen's picture

OK According to this you need to run compaction. You have 1.2 TB sitting there to be compacted.

Run /usr/openv/pdde/pdcr/bin/crcontrol --compactstart 0 0 1 (I believe)

The above comments are not to be construed as an official stance of the company I work for; hell half the time they are not even an official stance for me.

Mark_Solutions's picture

Thats what i was thinking - the good old compaaction issue!

You can also kick things off using crcontrol --dsstat 1

See this tech note (fixed in 7.5.0.4) : http://www.symantec.com/docs/TECH190714

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

Leo46's picture

I ran the compactstate but no items were compacted overnight.

crcontrol compactstart 0 0 1.PNG
captain jack sparrow's picture

Could you get --dsstat 1 output . till far it should have reclaimed partial or Full disk space for you. Was TN from mark referred if in case msdp pool has reached to 100%?

 Cheers !!!

CJS

 

Leo46's picture

Hi All,

 

 

We upgraded to Netbackup 7.5.0.4 yesterday but the "Orphaned containers" are still there at DR. PROD looks much better though but PROD was not the problem area to begin with. I did run a garbage collection after the upgrade but there is still a lot of already dereffrenced DATA in the MSDP pool wich hasnt cleared.

 

Marianne, any other suggestions possibly?

 

Thanks!

--dsstat.PNG