Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Altiris Recovery Solution - Space Issues

Created: 03 Feb 2010 • Updated: 20 Feb 2011 | 7 comments

I have a total of 25,000 users on 12 RS Cluster at about 2200 to 2300 users per cluster and 1 Notification server. I keep running out of space and a Full SSM's are taking longer to complete (36 to 40 days). I have an average of about 20TB of space per each cluster. We have lost close to 1000 employees and added a new VMAX Array SAN of 50TB and these clusters ran thru 50TB in 3 months, so i figure there is something wrong. Any guidance would be most appreciated..I have taken the steps below to try and eectify the issue but no success:

> Marked for deltion Duplicate and Terminated accounts>
> Ran D&C SMM
> Started a new process where I would run the Server Jobs in the order below and it has helped a little:
>>>>>>>>>> Full SSM next IC w\CRC next D&C SSM next Full SSM
> I have not check exclusion list and see whats being excluded (will need help with that)
> I have not checked the file extensions that are being copied that we may exclude (will need help with that)

Comments 7 CommentsJump to latest comment

KSchroeder's picture

Ariel,
Unfortunately that sounds about right, at least for the time required.  You really need to review your exclusions configuration and retention (particularly for Outlook PSTs).  We were suffering from similar issues (though not quite to that scale; we have a total of around 6000 users total).  We ultimately re-architected our environment and went for using numerous smaller VMs with < 500 clients each, and SSM completes within 24 hours typically.

What version of RS and hotfix are you on?  There are some issues with older versions and SSM functioning properly.  Even under RS 6.2 SP3 there is a hotfix where SSM can hang in certain situations.

Are all of your clusters single-node?  Again we're not doing this, but setting up multiple nodes per cluster (with one dedicated to running SSM for example) might help get the job done faster.  What size are your BLOB files?

Thanks,
Kyle
Symantec Trusted Advisor

For Forum threads, please click "Mark as Solution" if answered.
For all content, please give a thumbs up if you agree with or support the post.

KSchroeder's picture

Hi Ariel,
Any updates on your situation (or are you too busy trying to keep SSM running)?   I would again urge you to verify you're on the most current hotfixes for RS as there are critical fixes for SSM included in some of them.

Also...unless you're seeing issues restoring files and/or SSM is crashing due to corruption in the BLOB files, it is probably not necessary to run IC w/ CRC.

Thanks,
Kyle
Symantec Trusted Advisor

For Forum threads, please click "Mark as Solution" if answered.
For all content, please give a thumbs up if you agree with or support the post.

KSchroeder's picture

One other suggestion...you could try to get ahead of your SSM job a bit using AKB 41902 which allows you to tune how SSM/Deletion works.  We used to do this all the time to help the NS try to "catch up" with compaction.  You can run the following to check the progress of your SSM/Deletion job:

USE

AeXRSDatabase

SELECT '-- ' + cast(GetDate() as nvarchar(20)) AS [Current Date/Time],
    count (*) AS [Total Blocks],
    cast(sum(cast(BlockLength as numeric(20,2)) /(1024*1024)) as numeric(10,2)) AS [MB to Delete]
FROM DeletedBlobs

SELECT count (distinct FileKey) AS 'Dis. Filekeys', count (*) as 'total revisions' FROM Revision WHERE DeleteFlag = 1

This should help see how many blocks there are currently marked for deletion, as well as distinct files and total revisions.  This kind of helps gauge how close SSM is to being complete.

Thanks,
Kyle
Symantec Trusted Advisor

For Forum threads, please click "Mark as Solution" if answered.
For all content, please give a thumbs up if you agree with or support the post.

KSchroeder's picture

Ariel,
Any luck?  Have you been able to reclaim space?

Thanks,
Kyle
Symantec Trusted Advisor

For Forum threads, please click "Mark as Solution" if answered.
For all content, please give a thumbs up if you agree with or support the post.

pstage's picture

I had this same issue. From what I understand the delete marked items task is toward the end of the SSM task. If you are low on space, it never gets to this part. Run the Delete Marked Items job seperately first, then run the SSM job.

KSchroeder's picture

You can also control how SSM and Deletion work by setting the RetentionSettingsEx registry key:
AKB41902

Thanks,
Kyle
Symantec Trusted Advisor

For Forum threads, please click "Mark as Solution" if answered.
For all content, please give a thumbs up if you agree with or support the post.

Ariel KPMG's picture

Hi Kyle - I apologize for getting lost there for a second, I hope you understand i have been EXYTREMLY busy working late night hours and weekends trying to keep this environment up float. I have opened a case with the Symantec Help Desk in Estonia and i have been on and off with them for the past 3 months. It always seems we have found something but at the end no luck.. I am pretty familiar now with RS 3 SSM Jobs, Hotfixes, Etc.. Below is all we have done so far and we just can't get pass the hump of getting a fair amount of space back..

What has kept me up float has been running the exec get blobs com.sql and setting the Storage Compaction to where we get the most bang for our buck and making it where they can complete SSM (since it gives the space back at the end, if the SSM aborts or fails or we have to stop it we won't get anything back)  Example: I have 600GB of free space on a cluster hosting 2200 users with Snapshots turned off only on Wed, Sat & Sun OK.. If i run exec get blobs at 74% and it tells me it will compact 7800 blob files, that will make it 7800 x 250 (blob file size) = 1.9 TB of space to reclaim Note: i have 700GB to begin with. The server is taking in on a average 100GB of data a day. When this RS 3 complete techinicall i would end up with 1.5 and the RS 3 Job will wrap up in 4 to 6 days. This is good..Than i remove RS Reg Key, Restart ARS service and start Full SSM for marking. When i get low again 500GB to 600GB being my threshold i start aghain a RS3 Job calculating the Storage Compaction to a number where the RS 3 Job can finish and return space..Its a nightmare but at least is keeping me employed :-)

We have gone in and deleted Dead, Duplicate and Terminated user accounts on all clusters.
We have applied all SQL Monthly Maintenance Scripts (MigrateToVista, CleanUpScript & ExecOrphanVolumeCleanup)..
We have provided results for Support_LostBLOBsProcessing62sp2.sql, blobs2delete.txt & CalculateSpaceUsedForEachBlob.txt…
We have applied latest RS Hot Fixes (KB48955 & KB52365)
We have applied SSM performance improvement Stored Procedure (GetDelBlobRecs.sql & GetRevToRemove.sql)
We have Re-Index the Database to improve SSM Performance on all clusters.
We have sent Garbage Blob Query that was running for a week for analysis.
We have added 1 TB of free space to all clusters to improve Space Management Job.

I would like to discuss the idea of  numerous small VM's with maybe 1000 client each to see how quickly SSM can complete...I am aware and have been using most of the query's you mentioned and i have not ran a IC w\CRC on any of the clusters for the past 6 to 7 months and DMI jobs last to long and don't give much back..I would really hope you can find some time to possibly discuss this over the phone or even perform a remote control Session so that you may poke around this environment. It is a very large environment (i beleive the largest Symantec has) so its quite a challenge. I think the problem is with Retention on PST - I seriously believe we have tons of PST files that are sitting dead in the water that we cannnot get to in order to delete because the SSM's just have to much to do..I have attached a spreadsheet that contains the status of our clusters and the jobs we have been running for a quick birds eye view i gather...Anyway, Thank you for your time and patience on this matter..

AttachmentSize
Current Cluster Status Report.xls 831 KB