Video Screencast Help

Non-paged memory pool (memory leak)

Created: 29 Aug 2013 • Updated: 29 Aug 2013 | 22 comments

Here's the scenario:

 

Have 20+ domain controllers and from version 11 we upgrade them to 12.1 RU2.

One week or so, those regional servers started to cause a problem causing the users not to authenticate using their accounts as the event logs shows that the non-paged memory pool run out of allocated memory. As per checking, the servers cannot be mapped or browse the internet.. This happens to all domain controllers and stil lucky that it still not reaching the data center which will cause a lot of problem. Symantec Support says it is not cause by symantec? How would I know? because it never ever happened before until we upgrade those servers to 12.1 RU2. BTW, I provided the symantec Support a Symhelp.exe logs to check what causes the issue? Suggested to upgrade again to 12.1 RU3, but will it surely resolve the problem here?

 

 

 

Please help me as I'm still looking for another solution. This is not acceptable to the client and must be proactive monitoring the servers.

 

 

 

Thanks!

Ren

Operating Systems:

Comments 22 CommentsJump to latest comment

Rafeeq's picture

Stop all the Symantec releated services and check if issue persists. If it is , then ist Symantec. 

Do you have NTP component installed on DC?

Rafeeq's picture

Does it stop if you stop symantec services or rollback to previous version?

joash theory's picture

We have to disable first the auto protect to totally disable the SEP. But based on what I experience. It is still the same event ID 5719. non-paged memory pool due to insufficient memory allocation.

 

Previous version 11.0.7000.967

Latest version: 12.1.2015.2015

 

_Brian's picture

If you've disabled or removed SEP, than it isn't the likely cause.

Has anything else changed on the machine(s)?

Please click the "Mark as solution" link at bottom left on the post that best answers your question. This will benefit admins looking for a solution to the same problem.

Rafeeq's picture

It doesn't seem to related to SEP however its hard to convince microsoft if you have AV installed ;) 

P_K_'s picture

Can you plesae disable Tamper Protection and see if that helps.

If not take an output of netstat -ano and see if the server is running out of ports.

MCT MCSE-2012 Symantec Technical Specialist (SCTS)

P_K_'s picture

If that doesn't help we need take a Pefmon, Poolmon, Memory Dump from the srever to find the RCA of the issue.
 

MCT MCSE-2012 Symantec Technical Specialist (SCTS)

Mithun Sanghavi's picture

Hello,

I agree with the above comment. Could you please PM me your Case #?

Let me look into the case.

 

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

joash theory's picture

Hi Mithun,

 

I already PMed you my case id number for this issue.

 

 

Thanks!

joash theory's picture

Nothing changed on the server. all policies are intact no NTP installed only plain AV/AS only. Our Systems Administrator said that it never happened like this before, its just now that after the migration of SEP. No other softwares or other roles for this domain controllers. Basically, it happened now to 7 of our regional DCs. In past 2 weeks 7 of them causing this problems and since the last touch was done is SEP upgrade, the clients are suspecting SEP. But upon opening a case to Symantec Support, they said that it is not SEP which is kindly not clear as of of the moment. I already sent Symhelp logs but they see no cause of Symantec. Another support says that it would be better to send them also the Dump files for further investigation of the issue.

 

I thought that 12.1 RU2 has already fixed the memory leak problem but Symantec support Symhelp logs is not suffice to determine what really the cause of the issue. By the way, for the temporary resolution of the problem, the Admins restarted the servers in order to quickly restore the memory leak that causing the auhtentication problem of users. Admins are afraid that it would cause a bigger problem when this reaches their datacenters. All their systems, emails, firewalls and databases are all located there.

 

Now, as per the last task done by our admins they do server patching last sunday that causes all DC servers to restart one more time. We are waiting for the issue to rise again, waiting to see if the memory leak will still occurred and I hope it is not on Datacenter.

Beppe's picture

Dear Joash,

the SymHelp report is not a universal tool for every kind of IT issue. It is not designed to investigate on suspected memory leaks. PoolMon is the tool for such things, plus memory dump analysis:

http://support.microsoft.com/kb/177415/en-us

According to my experience, SEP 12.1 might cause high non-paged memory usage due to the fact that, even if NTP is not installed, the Symantec network driver is still loaded.

Try to disable the Symantec network driver (it is not necessary when the NTP is not installed) and let us know if you still see the same issue:

SC config <DriverName> start= disabled

Where <DriverName> is:

SymTDI: For Windows XP and  2003 servers
SymTDIv: For Windows Vista and 2008 (up to SP2) servers
SymNets: For Windows 7 and 2008 R2 servers
 
After the command completes successfully, the system should be restarted so the drivers can be disabled.
 
To restore functionality, type the following command then restart the system:
 
SC config <DriverName> start= system
 

You also need to distiguish a memory leak by "just" a high usage, it is not the same at all.

Regards,

Giuseppe

Beppe's picture

To be precise, the Support engineer sohuld see some memory usage report in the SymHelp report (yet not enough to spot a memory leak), but you can't see them by yourself without the SymHelp Viewer (Symantec internal only tool) hence, to deny that the issue is not caused by Symantec, you need to use Microsoft PoolMon tool and verify what process/driver is eating the memory.

Regards,

Giuseppe

joash theory's picture

Where is this SC config located? Ok, I think that's why also the Symantec Support asks for memory dump files....

Beppe's picture

SC is a Windows command...

Regards,

Giuseppe

joash theory's picture

ok let me not close this as we are still waiting for the problem to occur again.. right now, all DCs are normal and we are closely monitoring it.

 

 

Thanks for the suggestion and comments. Please do not close this issue yet, I really want to know the really cause of this issue.

 

 

Thank you.

Beppe's picture

When closing the thread is up to you...

Regards,

Giuseppe

joash theory's picture

ok. but even in SEP 12.1 RU3 it cannot resolve the problem , right?

Beppe's picture

Generally speaking, the upgrade is still suggested, however, a confirmation that it will help may come only once the investigation with the support on PoolMon outputs and memory dumps is completed.

Regards,

Giuseppe

picnic1234's picture

still having the leak? Just curious if you resolved that in your environment

Cheers,

 

joash theory's picture

The memory leak didnt occur anymore when the server was restarted by our systems admin. I guess what resolves to that is restart....