How to reclaim deduplication storage space manually (PureDisk Storage Pool and NetBackup Media Server Deduplication Pool)
| Article:TECH124914 | | | Created: 2010-01-04 | | | Updated: 2012-02-21 | | | Article URL http://www.symantec.com/docs/TECH124914 |
Problem
Users may want to reclaim deduplication storage space manually in PureDisk 6.6.x and NetBackup 6.5.x and 7.x environments.
To reclaim storage space manually on a NetBackup appliance, see http://www.symantec.com/docs/TECH180659.
Solution
The data removal process with NetBackup deduplication technologies (NetBackup Media Server Deduplication and PureDisk Deduplication) is different from the data removal process for standard NetBackup disk types (AdvancedDisk and BasicDisk).
With standard disk types there is a 1:1 mapping between the image record in the NetBackup catalog and the backup image on disk. When the image record expires, the backup image is removed.
With deduplication, as multiple references are linked to the same content, the removal process is run in multiple transactions, leading to a delay between the expiration of the backup image and the storage reclamation. Note that in larger production environments, this short delay should not be an issue and should be irrelevant.
NetBackup provides two deduplication products:
- Symantec NetBackup PureDisk is a product that provides a complete backup and deduplication environment. It can be used as a stand-alone (PureDisk writes data to the storage pools) or as a back-end for NetBackup.
- NetBackup 7.0 and later delivers deduplication integrated into NetBackup at the media server and supports a dedicated deduplication pool called a Media Server Deduplication Pool (MSDP).
Under normal operation, a sequence of regularly scheduled operations ensures that obsolete data segments are removed from the storage pools or deduplication pools automatically. Because each data segment may be owned by more than one backup image, segments are only removed when all the associated backups have expired.
This document applies to PureDisk 6.6.x and NetBackup 6.5.x and later and outlines the steps required to expire the backups at the NetBackup level and then release the space in the deduplication pool. It can help you to understand and possibly accelerate the removal of deduplicated data a little.
The crcollect and crcontrol commands
The crcollect and crcontrol commands are required for manual operations; these commands are located in the following directories:
- On a PureDisk Storage Pool server - /opt/pdcr/bin
- On a UNIX or Linux MSDP Media Server - /usr/openv/pdde
- On a Windows MSDP media server - <install path>\pdde
Display the storage statistics
To display the storage statistics, (the operating system commands do not provide an accurate measurement of deduplication storage), use the PureDisk crcontrol --dsstat command on the MSDP media server or the PureDisk Storage Pool server to show the storage statistics, as in the following example.
% crcontrol --dsstat
************ Data Store statistics ************
Data storage Size Used Avail Use%
35.1T 5.3T 29.8T 15%
Number of containers : 98125
Average container size : 268211087 bytes (255.79MB)
Space allocated for containers : 26318212931722 bytes (23.94TB)
Space used within containers : 24563636296449 bytes (22.34TB)
Space available within containers: 1754576635273 bytes (1.60TB)
Space needs compaction : 18875042669860 bytes (17.17TB)
Records marked for compaction : 139455139
Active records : 43562983
Total records : 183018122
==
Data storage Size Used Avail Use%
35.1T 5.3T 29.8T 15%
Number of containers : 98125
Average container size : 268211087 bytes (255.79MB)
Space allocated for containers : 26318212931722 bytes (23.94TB)
Space used within containers : 24563636296449 bytes (22.34TB)
Space available within containers: 1754576635273 bytes (1.60TB)
Space needs compaction : 18875042669860 bytes (17.17TB)
Records marked for compaction : 139455139
Active records : 43562983
Total records : 183018122
==
The following figure is a visual representation of the storage space:
From the crcontrol --dsstat output, you can determine the following:
- Total size is obtained from the operating system, 35.1T in the preceding example output.
- The total used space = the file system used space - space available within containers - space needs compaction.
For the preceding example output, X - 1.6TB - 17.17TB = 5.3TB. NetBackup / PureDisk obtains the file system used space and uses that value for X. - Available space = the total size - used space, 35.1TB - 5.3TB = 29.8TB in this example.
- The percentage used is the used space / total size, 5.3TB / 35.1TB = 15%.
- The space actually used to store data = the space used within containers - the space that needs compaction: 22.34TB - 17.17TB = 5.17TB. This value differs from the 5.3TB in the example output because the command output includes overhead space that is used in the data store.
About space compaction
NetBackup and PureDisk compact the free space that is created in the data store when the data fragments expire. The compaction process runs when storage is required or when the system is idle; it reorganizes the storage by physically deleting the dereferenced empty blocks from the file system and compacts the storage containers to the most optimal format. Because compaction occurs frequently, you do not need to compact space manually.
To check if compaction is configured properly, run the following command:
crcontrol --compactstate
It should return output similar to the following:
Data store compaction: ON, DeleteSpaceThreshold: 30%, CompactLBound: 4MB
Compaction busy: No [or Yes]
Why space is not reclaimed
Some reasons why space may not be reclaimed are as follows:
- Queue processing is not running or failing. Check your deduplication activity logs.
- PDDO data removal is not running or failing (PureDisk storage pools only).
- Content router modes disallow deref and delete operations (PureDisk storage pools only, after rerouting of Disaster Recovery Backup jobs). To check and set content router modes, see the Symantec NetBackup PureDisk Administrator's Guide.
- Compaction is disabled or the CompactWakeUpInterval is set too high.
How NetBackup and PureDisk reclaim space
Reclamation of the redundant storage is driven by the user's configured retention settings in the backup policy. NetBackup's deduplication technologies automatically reclaim storage from the container files by following a specific sequence of transactions. (Once a container file is used, it is managed by the deduplication store and will not be released to the file system.) This sequence is slightly different for the following 3 use cases:
Use case 1: Data protected with a NetBackup client and stored on a Media Server Deduplication Pool (MSDP).
- Based on the configured retention, NetBackup expires the images in the NetBackup image catalog (image cleanup job) and then creates corresponding expiration transactions in the deduplication transaction queue.
- Transactions in the queue are processed by the deduplication daemons every 12 hours, 20 minutes past the hour, and the corresponding data pointers are dereferenced from storage.
- A garbage collection runs weekly to clean up any orphaned data, which can occur in certain edge cases.
Use case 2: Data protected with a NetBackup client and stored on an external PureDisk Storage Pool.
- Based on the configured retention, NetBackup expires the images in the NetBackup image catalog (image cleanup job). NetBackup also notifies the PureDisk MetaBase (catalog) of the images to be deleted.
- On a weekly basis the PDDO Data Removal Policy removes the corresponding NetBackup image data from the PureDisk MetaBase engine (catalog) and creates corresponding expiration transactions in the PureDisk deduplication transaction queue. The schedule of the PDDO Data Removal Policy can be changed in the PureDisk WebUI to daily if needed.
- Transactions in the queue are processed by the Queue Processing Policy, and the corresponding data pointers are dereferenced from storage. The schedule of the Queue Processing is 2 times a day by default; it can be changed from the PureDisk WebUI.
- On a monthly basis a CR garbage collection policy runs automatically to clean up any orphaned data, which can occur in certain edge cases. If you have very short retention policies (retention of less than 5 days), then it is useful to schedule CR Garbage Collection more frequently (once or twice a week).
Use case 3: Data protected with a PureDisk client and stored on a PureDisk Storage Pool.
- The data removal policies, which run at a user's scheduled time, automatically dereference the affected files in the MetaBase Engine (catalog) and add the necessary expiration transactions in the Content Router (CR) queue for removal.
- Transactions in the queue are processed by the Queue Processing Policy, and the corresponding data pointers are dereferenced from storage. The schedule of the Queue Processing is 2 times a day by default; it can be changed from the PureDisk WebUI.
- On a monthly basis a CR garbage collection policy runs automatically to clean up any orphaned data, which can occur in certain edge cases. If you have very short retention policies (retention of less than 5 days), then it is useful to schedule CR Garbage Collection more frequently (once or twice a week).
Important note about multi-node PureDisk Storage Pools
The following procedures will not have the same predictable effect for PureDisk multi-node storage pools. A multi-node storage pool has additional retention mechanisms built in to coordinate the protection of data against premature removal across all nodes. This complex removal process makes it difficult reclaim space faster than PureDisk reclaims space automatically. It would require you to synchronously process the queues of all nodes until they are eventually empty, to wait a few days, run CR Garbage Collection, and process the transaction queue two more times.
How to reclaim space more quickly
To reclaim space more quickly, do the following:
- Identify what to expire: NetBackup backup images (use case 1 and 2) or PureDisk backup images (use case 3).
- Expire backup images that are no longer required. When you expire images in either NetBackup or PureDisk, transactions that remove the deduplicated image fragments are then generated and placed in the deduplication transaction queue (either NetBackup (use case 1) and/or PureDisk (use cases 2 and 3).
- See Expire NetBackup images (use cases 1 and 2 only)
- See Expire PureDisk images in the metabase (use case 3 only)
- Process the transaction queue twice so that unnecessary fragments are removed from storage.
- See Process the transaction queue.
You can also remove garbage data at any time. If you remove garbage data, you must also process the transaction queue twice. Any time that you do something manually that generates transactions, you should process the transaction queue twice to complete the operation. The crcollect and crcontrol commands are required for these manual operations. See Remove garbage data.
Expire NetBackup images (use cases 1 and 2 only)
Expiring NetBackup images clean up all image references in the NetBackup catalog and EMM database.
1. Expire the backup images by using the NetBackup GUI or the command line.
GUI initiated expiration:
Under Catalog, search for all existing images stored on the Media Server Deduplication pool. Make sure you search across the correct data range. Once found, right click and expire all images. You can check the Activity Monitor to verify if the Image Cleanup jobs succeeded successfully. This removal process dereferences the images from the NetBackup catalog.
Command line driven expiration:
a) Use the bpimagelist command to determine the backup IDs of the backups to be expired.
b) Run the command bpexpdate -backupid <backip ID> -d 0 -force -notimmediate to expire each image. The -notimmediate option prevents bpexpdate from calling the nbdelete command, which deletes the image. Without this option, bpexpdate calls nbdelete to delete images. Each call to nbdelete creates a job in the Activity Monitor, allocates resources, and launches processes on the media server.
c) After you expire the last image, delete all of the images by using the nbdelete command with the -allvolumes option. Only one job is created in the Activity Monitor, fewer resources are allocated, and fewer processes are started on the media servers. The entire process of expiring images and deleting images takes less time.
2. If the backup images are in a PureDisk Storage Pool (use case 2 only), run the PureDisk data removal policy in the PureDisk web based interface.
3. Process the transaction queue twice. See Process the transaction queue.
Expire PureDisk images in the metabase (use case 3 only)
Follow these steps for PureDisk (use case 3 only). These steps are not necessary for data protected with the NetBackup clients.
-
Create (if necessary) and run a data removal policy. Configure a removal policy to define what versions you would like to remove. Select a data retention in days or select a version retention to be more specific.
- Process the transaction queue twice. See Process the transaction queue.
Remove garbage data
In a few rare scenarios, some data segments may become orphaned. Garbage collection cleans these segments up by removing them.
- Run the following command:
crcollect -v -m +1,+2 - Process the transaction queue twice. See Process the transaction queue.
Process the transaction queue
During the backup process, transactions are stored in a transaction queue. The transactions are located in a queue directory under the deduplication storage path. For the PureDisk server, the storage path is /Storage; for the MSDP server the path is customized during the installation. The queue directory contains files with the extensions .tlog and .delayed.
The backup jobs, queue processing, and many other operations can generate new entries in the queue. Therefore, when you process the queue, you will notice that some of the old .tlog files disappear, but new .tlog files are generated.
After one queue processing is finished, the oldest .tlog file should not be older than the start time of that last queue processing. Otherwise, contact support and provide the storaged.log.
To process the transaction queue from the command line on the MSDP media server or PureDisk Storage Pool server, perform the following steps:
- Check if another queue processing is running:
crcontrol --processqueueinfo - Repeat step 1 until pending and busy status show "no" as the result.
Remark: one queue processing can take between a few minutes up to about a day. Progress can be seen in the storaged.log. - Start the CR queue processing:
crcontrol --processqueue - Check if the queue processing is finished by using the following command:
crcontrol --processqueueinfo - Repeat step 4 until pending and busy status show "no" as the result.
The storaged.log is located in the following directories:
- On a PureDisk Storage Pool server - /Storage/log/spoold/storaged.log
- On a UNIX or Linux MSDP Media Server - <storage path>/log/spoold/storaged.log
- On a Windows MSDP media server - <storage path>\log\spoold\storaged.log
|
|
Legacy ID
346797
Article URL http://www.symantec.com/docs/TECH124914
Terms of use for this information are found in Legal Notices
Thank you.