Video Screencast Help
Search Video Help Close Back
to help
Not able to make it to Vision this year? Get a sampling in the Best of Vision on Demand group.

SEP 11 MR4 MP2 - Scheduled scan cause high pool paged memory use on Citrix Servers

Updated: 04 Oct 2010 | 31 comments
ernieken's picture
0 0 Votes
Login to vote

We notice that the weekly scheduled scan on our Citrixservers causes excessive use of the paged pool memory on our Citrixservers. The pool page remains high until the server is rebooted. Until than the server is quite irresponsive. See graph below.
memory-pool paged bytes.jpg

Any idea what we can do about this? Cannot find some useful information on the internet. Servers are W2K3 SP2 with 8GB RAM.

Discussion Filed Under:

Comments

AravindKM's picture
13
Oct
2009
0 Votes 0
Login to vote

Try this

 

Try this

Best Practices for Symantec Endpoint Protection on Citrix
and Terminal Servers

http://service1.symantec.com/SUPPORT/ent-security.nsf/2326c6a13572aeb788257363002b62aa/f5e1baf6ca2b5d638025750b00511265?OpenDocument

Please don't forget to mark your thread solved with whatever answer helped you : ) Thanks & Regards Aravind

ernieken's picture
13
Oct
2009
0 Votes 0
Login to vote

We already applied the best

We already applied the best practices for SEP on Citrix server following the document mentioned above.
The high pool page memory use is only during the scheduled scan. During realtime scan we do not see this behaviour.

AravindKM's picture
13
Oct
2009
0 Votes 0
Login to vote

Confirm following exception

Confirm following exception
are made and it is got affected in your server

Centralized Exceptions

It is recommended to:

· Exclude the pagefile

· Exclude the print
spooler folder

· If the server is a
license server, exclude the license server folder and databases

 

for this you can refer the
below doc

How to Verify if an Endpoint
Client has Automatically Excluded an Application or Directory

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2008090512574448?Open&seg=ent

 

Please don't forget to mark your thread solved with whatever answer helped you : ) Thanks & Regards Aravind

Eduardo Menegalli Nazato's picture
13
Oct
2009
0 Votes 0
Login to vote

In theory, the pagefile

In theory, the pagefile should be excluded by default from scans, right?

AravindKM's picture
13
Oct
2009
0 Votes 0
Login to vote

If above solution is not

If above solution is not working try by
upgrading it to RU5.There is a lot of fixes related to scanning is present in
RU5.

ref: Release notes of RU5

 

Please don't forget to mark your thread solved with whatever answer helped you : ) Thanks & Regards Aravind

ernieken's picture
15
Oct
2009
0 Votes 0
Login to vote

Pagefile is excluded.

Pagefile is excluded. Installing RU5 can be a solution. Maybe we can try that.

AravindKM's picture
15
Oct
2009
0 Votes 0
Login to vote

It is better to install only

It is better to install only
av/as ........

All the best......

 

 

Please don't forget to mark your thread solved with whatever answer helped you : ) Thanks & Regards Aravind

ernieken's picture
04
Dec
2009
0 Votes 0
Login to vote

Nobody with a similar

Nobody with a similar situation?
Because I think this is really a bug. When I start a full scan on a Win XP SP3 workstation I see an increase of the paged pool memory during the scan of about 40MB (depending on the number of files and compressed files) and the paged pool memory isn't released after a scan. For a normal workstation that's not too bad because you also boot the system from time to time but in our terminal server environment this is a real issue because the number of concurrent users on one Citrix server is limited to 7 (where 20 should be possible).

In the meanwhile I excluded lots of things (bloodhound detection etc.) but without success.
Case is now with Symantec. I'll hope they can find something.

Rafeeq's picture
04
Dec
2009
0 Votes 0
Login to vote

Hi

Hi

Did you upgrade to mu5
what you said was correct it was a bug  check this for more info

High paged pool memory usage for Auto-Protect
Fix ID: 1511152
Symptom: Pool monitor shows high memory usage for SavE and SaEe pooltags.
Solution: AV engine update.

 

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007121216360648

 

Please don't forget to mark your thread solved with whatever answer helped you : ) Rafeeq

Christoph_A's picture
04
Dec
2009
0 Votes 0
Login to vote

You were not alone!

You were not alone!
We have the same issue since a few weeks. A lot of citrix servers ran into the same problem.
Currently 10 out of 80 servers have that issue. This is more than 10%.We have applied RU5 but this does not solve the problem.
I think this is really a BUG!

We moved the scheduled scan to the weekend. The servers were rebooted before and now it works a little better. But only a little.

Regards,
Christoph

ernieken's picture
04
Dec
2009
0 Votes 0
Login to vote

We already deployed RU5 to

We already deployed RU5 to our Citrix Servers but that doesn't change much (or even nothing).
Rebooting clears indeed the paged pool memory but the start level is already high (110MB). After the the initial installation of SEP this was only 80MB. We also noticed that the paged pool memory rose a lot after new virus definitions were loaded.
We have also Appsense and AppV running on our Citrixservers.

AravindKM's picture
04
Dec
2009
0 Votes 0
Login to vote

Edit antivrus and antivirus

Edit antivrus and antivirus policy go to administrator defined scans---->advanced remove the options run startup scan when user logged on and  run active scan when new definition arrives and see any difference is present....

Please don't forget to mark your thread solved with whatever answer helped you : ) Thanks & Regards Aravind

ernieken's picture
04
Dec
2009
0 Votes 0
Login to vote

Startup scan and active scan

Startup scan and active scan when new definitions arrive was already excluded.

Rafeeq's picture
04
Dec
2009
0 Votes 0
Login to vote

Hi

 Please disable file system autoprotected and check the performance, by doing so we can narrow down to the root cause. if its with autoprotect

Please don't forget to mark your thread solved with whatever answer helped you : ) Rafeeq

ernieken's picture
04
Dec
2009
0 Votes 0
Login to vote

Disabling autoprotect doesn't

Disabling autoprotect doesn't free up pool paged memory.
To be clear: system becomes slow to irresponsive as soon as pool paged memory rises over a certain limit (+300MB).
After the scheduled scan the pool paged memory is very high and remains high till the system is rebooted.

Christoph_A's picture
04
Dec
2009
0 Votes 0
Login to vote

Same to us. Startup scan is

Same to us.

Startup scan is disabled.
Are you sure that it makes sense to disable autoprotect during the scheduled scan? sure?

sezam's picture
04
Dec
2009
0 Votes 0
Login to vote

  Start Registry Editor

 

  1. Start Registry Editor (Regedt32.exe).
  2. Locate and then click the following key in the registry:
    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management
  3. On the Edit menu, click Add Value, and then add the following registry value:
    Value name: PoolUsageMaximum
    Data type: REG_DWORD
    Radix: Decimal
    Value data: 60

    Setting the value at 60 informs the Memory Manager to start the trimming process at 60 percent of PagedPoolMax rather than the default setting of 80 percent. If a threshold of 60 percent is not enough to handle spikes in activity, reduce this setting to 50 percent or 40 percent.

    Value name: PagedPoolSize
    Data type: REG_DWORD
    Radix: Hex
    Value data: 0xFFFFFFFF

    Setting PagedPoolSize to 0xFFFFFFFF allocates the maximum paged pool in lieu of other resources to the computer.

    Caution The 0xFFFFFFFF PagedPoolSize setting is not recommended for use on 32-bit Windows Server 2003-based computers that have 64GB of RAM. This will potentially bring the Free System PTE entry down and can cause continuous reboot of the computer. For this configuration, carefully choose a value based on the requirements and available resources.

  4. Quit Registry Editor.
  5. Restart the server for the changes to take effect.
ernieken's picture
06
Dec
2009
0 Votes 0
Login to vote

I agree that this registry

I agree that this registry hack will make the server more responsive when pool paged memory consumption is very high but I suppose this is in fact a workaround because this hack doesn't change the fact that:

  • Paged pool memory is high with SEP enabled
  • Paged pool memory isn't realeased after doing a scheduled scan

Or am I wrong?

AravindKM's picture
06
Dec
2009
0 Votes 0
Login to vote

I think it is better to open

I think it is better to open a case with symatec support. Once the problem got rectified pls post it here. 

Please don't forget to mark your thread solved with whatever answer helped you : ) Thanks & Regards Aravind

ernieken's picture
07
Dec
2009
0 Votes 0
Login to vote

Case already opened with

Case already opened with Symantec support. ;-)
I'm curious to hear if they have a miracle solution.

ernieken's picture
10
Dec
2009
0 Votes 0
Login to vote

What we see now is that when

What we see now is that when new virusdefinitons are loaded the paged pool memory increases dramatically after a boot. When we boot again than paged pool decreases again.
This becomes blocking for us. I hope Symantec will solve this issue very soon.
Anybody who noticed this also.

Rafeeq's picture
10
Dec
2009
0 Votes 0
Login to vote

Hi

what i can think of is manager sending out the definitions
if you have livedupate administrator installed.
make clients to get the definitions from luadmin
using san?
are these vmwares?

check this link

http://communities.vmware.com/message/1210918;jsessionid=27730EC892B2D572D41A8078E9EA7242
Courtesy: sandeep cheema.

Please don't forget to mark your thread solved with whatever answer helped you : ) Rafeeq

ernieken's picture
10
Dec
2009
0 Votes 0
Login to vote

Additional info: during scan

Additional info: during scan the MmSt pool tag is increasing.
The SaEe pooltag doesn't change.
So the trigger is from Symantec but in fact is not the Symantec SaEe tag that's increasing.

ernieken's picture
21
Dec
2009
0 Votes 0
Login to vote

Additional info: On a clean

Additional info:
On a clean install of RU5 it's better: during the scan the paged pool memory increases but when a certain treshold is reached paged pool memory is being released. When the scan is done the use of paged pool memory is acceptable but still not as low as before the scan.
So here my question is: what's the difference in settings between an clean install and an upgrade. Is there somewhere a registry setting we can adapt so SEP triggers the memory manager to free up paged pool memory (as happens with a clean RU5 install).
We cannot completely uninstall SEP and reinstall again SEP on all our servers to partially resolve this issue.
Symantec Support is examining my case but up to now I haven't received any valuable information from them.

ernieken's picture
24
Dec
2009
0 Votes 0
Login to vote

Update: in the meanwhile I

Update: in the meanwhile I discovered that the SaEe pooltag was taking about 55MB of poolpaged RAM and it should be only 35MB. This is due to an engine update somewhere between april 2009 and september 2009.
Test done: clean install of SEP MR4 MP2 with original defs (april 2009). Pooltag SaEe about 35MB. After updating definitions 55MB
Second test: clean install of RU5 with original defs (sep 2009) Pooltag SaEe about 55MB
Third test: clean install of RU5 with definitions of MR4 MP2 package (april 2009): Pooltag SaEe about 35MB. After updating definitions 55MB

Issue escaleted towards Symantec support and they told me to examine this issue with the highest priority.

Aniket Amdekar's picture
24
Dec
2009
0 Votes 0
Login to vote

Hay, Very neat info in the

Hay,

Very neat info in the test results. Keep us posted about what happens in the support case you have opened up.

Cheers,
Aniket

ernieken's picture
04
Jan
2010
0 Votes 0
Login to vote

Update coming from a Symantec

Update coming from a Symantec employee:

The engine has been updated and targeted to be released in January, Please find below the initial tests results.
 
Signatures Early 2009 :                   35MB of Paged Pool
Signatures End 2009 :                     54MB of Paged Pool
Signatures January 2010 :             40MB of Paged Pool

It seemed to be a 'bug'.
So I'm waiting for the new january signatures that will solve one part of our problem.

Jay Long's picture
21
Jan
2010
0 Votes 0
Login to vote

Hi, Has the update arrived

Hi,

Has the update arrived yet? If so has it made any difference?

I've spent a load of time tracking down this issue as the issue coincided with a Citrix migration to a later version. The new farm wouldn't accept as many user connections as the old one. After raising many support calls and sending dump files off to Citrix they pointed out that SEP was using a high amount of Paged-Pool area. Removing SEP allowed additional user sessions on the servers (not as many as we'd like but it was an improvement!)

Thanks

ernieken's picture
21
Jan
2010
0 Votes 0
Login to vote

The new engine that uses 15MB

The new engine that uses 15MB less pool paged memory are scheduled for the beginning of February 2010.
We have calls open with Microsoft and Symantec but up to now they haven't found anything interesting.
After implementing a lot of hotfixes (http://blogs.technet.com/yongrhee/archive/2009/09/14/list-of-terminal-services-related-hotfixes-for-post-service-pack-2-for-windows-server-2003.aspx) we saw an improvement.
We also raised the pool paged memory level to the maximum and we trim the paged pool memory at 60%.
I've noticed that a clean install of W2K3 and SEP does not give an inrease of the paged pool memory during the scan. So we are planning to create a clean system with SEP and add components step by step to see which component is causing the problem.
If you have some other interesing information, don't hesitate to post them here. I'm glad that we aren't the only ones with issues for the Citrix/SEP combination.

Jay Long's picture
01
Apr
2010
0 Votes 0
Login to vote

So far...

I haven't increased the paged pool settings from default. I'm concerned that this may cause other instability issues from something listed in an MS article. Did you increase the values to the maximum?

We are using the /PAE switch on our servers,  not the /3GB which will actually decrease the amount of paged pool memory available. How I wish the applications we publish were supported on a 64-bit O/S, would save me having to build extra Citrix servers.

teiva-boy's picture
03
Apr
2010
0 Votes 0
Login to vote

I'm curious as I've seem some

I'm curious as I've seem some paging issues with the cache function in SEP...  This is somewhere buried in the AV/AS policy where it caches files that it has already scanned (Default value is 10,000, turn it off instead).  Supposed to increase performance, but at the cost of memory..  In <1% of machines I've worked with (and that is in the thousands) it solved some weird performance issues.

There is an online portal, save yourself the long hold times. Create ticket online, then call in with ticket # in hand :-) http://mysupport.symantec.com "We backup data to restore, we don't backup data just to back it up."