Endpoint Protection

We notice that the weekly scheduled scan on our Citrixservers causes excessive use of the paged pool memory on our Citrixservers. The pool page remains high until the server is rebooted. Until than the server is quite irresponsive. See graph below.


Any idea what we can do about this? Cannot find some useful information on the internet. Servers are W2K3 SP2 with 8GB RAM.

 

Try this

Best Practices for Symantec Endpoint Protection on Citrix and Terminal Servers

http://service1.symantec.com/SUPPORT/ent-security.nsf/2326c6a13572aeb788257363002b62aa/f5e1baf6ca2b5d638025750b00511265?OpenDocument

We already applied the best practices for SEP on Citrix server following the document mentioned above.
The high pool page memory use is only during the scheduled scan. During realtime scan we do not see this behaviour.

Confirm following exception are made and it is got affected in your server

Centralized Exceptions

It is recommended to:

· Exclude the pagefile

· Exclude the print spooler folder

· If the server is a license server, exclude the license server folder and databases

for this you can refer the below doc

How to Verify if an Endpoint Client has Automatically Excluded an Application or Directory

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2008090512574448?Open&seg=ent

 

In theory, the pagefile should be excluded by default from scans, right?

If above solution is not working try by upgrading it to RU5.There is a lot of fixes related to scanning is present in RU5.

ref: Release notes of RU5

 

Pagefile is excluded. Installing RU5 can be a solution. Maybe we can try that.

It is better to install only av/as ........

All the best......

 

Nobody with a similar situation?
Because I think this is really a bug. When I start a full scan on a Win XP SP3 workstation I see an increase of the paged pool memory during the scan of about 40MB (depending on the number of files and compressed files) and the paged pool memory isn't released after a scan. For a normal workstation that's not too bad because you also boot the system from time to time but in our terminal server environment this is a real issue because the number of concurrent users on one Citrix server is limited to 7 (where 20 should be possible).

In the meanwhile I excluded lots of things (bloodhound detection etc.) but without success.
Case is now with Symantec. I'll hope they can find something.

Hi

Did you upgrade to mu5
what you said was correct it was a bug  check this for more info

High paged pool memory usage for Auto-Protect
Fix ID: 1511152
Symptom: Pool monitor shows high memory usage for SavE and SaEe pooltags.
Solution: AV engine update.

 

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007121216360648

 

You were not alone!
We have the same issue since a few weeks. A lot of citrix servers ran into the same problem.
Currently 10 out of 80 servers have that issue. This is more than 10%.We have applied RU5 but this does not solve the problem.
I think this is really a BUG!

We moved the scheduled scan to the weekend. The servers were rebooted before and now it works a little better. But only a little.


Regards,
Christoph

We already deployed RU5 to our Citrix Servers but that doesn't change much (or even nothing).
Rebooting clears indeed the paged pool memory but the start level is already high (110MB). After the the initial installation of SEP this was only 80MB. We also noticed that the paged pool memory rose a lot after new virus definitions were loaded.
We have also Appsense and AppV running on our Citrixservers.

Edit antivrus and antivirus policy go to administrator defined scans---->advanced remove the options run startup scan when user logged on and  run active scan when new definition arrives and see any difference is present....

Startup scan and active scan when new definitions arrive was already excluded.

 Please disable file system autoprotected and check the performance, by doing so we can narrow down to the root cause. if its with autoprotect

Disabling autoprotect doesn't free up pool paged memory.
To be clear: system becomes slow to irresponsive as soon as pool paged memory rises over a certain limit (+300MB).
After the scheduled scan the pool paged memory is very high and remains high till the system is rebooted.

Same to us.

Startup scan is disabled.
Are you sure that it makes sense to disable autoprotect during the scheduled scan? sure?

 

1. Start Registry Editor (Regedt32.exe).
2. Locate and then click the following key in the registry:
   HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management
3. On the Edit menu, click Add Value, and then add the following registry value:
   Value name: PoolUsageMaximum
   Data type: REG_DWORD
   Radix: Decimal
   Value data: 60
   Setting the value at 60 informs the Memory Manager to start the trimming process at 60 percent of PagedPoolMax rather than the default setting of 80 percent. If a threshold of 60 percent is not enough to handle spikes in activity, reduce this setting to 50 percent or 40 percent.
   Value name: PagedPoolSize
   Data type: REG_DWORD
   Radix: Hex
   Value data: 0xFFFFFFFF
   Setting PagedPoolSize to 0xFFFFFFFF allocates the maximum paged pool in lieu of other resources to the computer.
   
   Caution The 0xFFFFFFFF PagedPoolSize setting is not recommended for use on 32-bit Windows Server 2003-based computers that have 64GB of RAM. This will potentially bring the Free System PTE entry down and can cause continuous reboot of the computer. For this configuration, carefully choose a value based on the requirements and available resources.
4. Quit Registry Editor.
5. Restart the server for the changes to take effect.

I agree that this registry hack will make the server more responsive when pool paged memory consumption is very high but I suppose this is in fact a workaround because this hack doesn't change the fact that:

• Paged pool memory is high with SEP enabled
• Paged pool memory isn't realeased after doing a scheduled scan

Or am I wrong?

I think it is better to open a case with symatec support. Once the problem got rectified pls post it here. 

Case already opened with Symantec support. ;-)
I'm curious to hear if they have a miracle solution.

What we see now is that when new virusdefinitons are loaded the paged pool memory increases dramatically after a boot. When we boot again than paged pool decreases again.
This becomes blocking for us. I hope Symantec will solve this issue very soon.
Anybody who noticed this also.

what i can think of is manager sending out the definitions
if you have livedupate administrator installed.
make clients to get the definitions from luadmin
using san?
are these vmwares?

check this link

http://communities.vmware.com/message/1210918;jsessionid=27730EC892B2D572D41A8078E9EA7242
Courtesy: sandeep cheema.

Additional info: during scan the MmSt pool tag is increasing.
The SaEe pooltag doesn't change.
So the trigger is from Symantec but in fact is not the Symantec SaEe tag that's increasing.

Additional info:
On a clean install of RU5 it's better: during the scan the paged pool memory increases but when a certain treshold is reached paged pool memory is being released. When the scan is done the use of paged pool memory is acceptable but still not as low as before the scan.
So here my question is: what's the difference in settings between an clean install and an upgrade. Is there somewhere a registry setting we can adapt so SEP triggers the memory manager to free up paged pool memory (as happens with a clean RU5 install).
We cannot completely uninstall SEP and reinstall again SEP on all our servers to partially resolve this issue.
Symantec Support is examining my case but up to now I haven't received any valuable information from them.

Update: in the meanwhile I discovered that the SaEe pooltag was taking about 55MB of poolpaged RAM and it should be only 35MB. This is due to an engine update somewhere between april 2009 and september 2009.
Test done: clean install of SEP MR4 MP2 with original defs (april 2009). Pooltag SaEe about 35MB. After updating definitions 55MB
Second test: clean install of RU5 with original defs (sep 2009) Pooltag SaEe about 55MB
Third test: clean install of RU5 with definitions of MR4 MP2 package (april 2009): Pooltag SaEe about 35MB. After updating definitions 55MB

Issue escaleted towards Symantec support and they told me to examine this issue with the highest priority.

Hay,

Very neat info in the test results. Keep us posted about what happens in the support case you have opened up.

Cheers,
Aniket

Update coming from a Symantec employee:

It seemed to be a 'bug'.
So I'm waiting for the new january signatures that will solve one part of our problem.

Hi,

Has the update arrived yet? If so has it made any difference?

I've spent a load of time tracking down this issue as the issue coincided with a Citrix migration to a later version. The new farm wouldn't accept as many user connections as the old one. After raising many support calls and sending dump files off to Citrix they pointed out that SEP was using a high amount of Paged-Pool area. Removing SEP allowed additional user sessions on the servers (not as many as we'd like but it was an improvement!)

Thanks

The new engine that uses 15MB less pool paged memory are scheduled for the beginning of February 2010.
We have calls open with Microsoft and Symantec but up to now they haven't found anything interesting.
After implementing a lot of hotfixes (http://blogs.technet.com/yongrhee/archive/2009/09/14/list-of-terminal-services-related-hotfixes-for-post-service-pack-2-for-windows-server-2003.aspx) we saw an improvement.
We also raised the pool paged memory level to the maximum and we trim the paged pool memory at 60%.
I've noticed that a clean install of W2K3 and SEP does not give an inrease of the paged pool memory during the scan. So we are planning to create a clean system with SEP and add components step by step to see which component is causing the problem.
If you have some other interesing information, don't hesitate to post them here. I'm glad that we aren't the only ones with issues for the Citrix/SEP combination.

I haven't increased the paged pool settings from default. I'm concerned that this may cause other instability issues from something listed in an MS article. Did you increase the values to the maximum?

We are using the /PAE switch on our servers,  not the /3GB which will actually decrease the amount of paged pool memory available. How I wish the applications we publish were supported on a 64-bit O/S, would save me having to build extra Citrix servers.

I'm curious as I've seem some paging issues with the cache function in SEP...  This is somewhere buried in the AV/AS policy where it caches files that it has already scanned (Default value is 10,000, turn it off instead).  Supposed to increase performance, but at the cost of memory..  In <1% of machines I've worked with (and that is in the thousands) it solved some weird performance issues.