Endpoint Protection

Here's the scenario:

Have 20+ domain controllers and from version 11 we upgrade them to 12.1 RU2.

One week or so, those regional servers started to cause a problem causing the users not to authenticate using their accounts as the event logs shows that the non-paged memory pool run out of allocated memory. As per checking, the servers cannot be mapped or browse the internet.. This happens to all domain controllers and stil lucky that it still not reaching the data center which will cause a lot of problem. Symantec Support says it is not cause by symantec? How would I know? because it never ever happened before until we upgrade those servers to 12.1 RU2. BTW, I provided the symantec Support a Symhelp.exe logs to check what causes the issue? Suggested to upgrade again to 12.1 RU3, but will it surely resolve the problem here?

Please help me as I'm still looking for another solution. This is not acceptable to the client and must be proactive monitoring the servers.

Thanks!

Ren

hi,

any solutions or comments or suggestions?

Stop all the Symantec releated services and check if issue persists. If it is , then ist Symantec. 

Do you have NTP component installed on DC?

Actually, only AV/AS are installed.

Does it stop if you stop symantec services or rollback to previous version?

We have to disable first the auto protect to totally disable the SEP. But based on what I experience. It is still the same event ID 5719. non-paged memory pool due to insufficient memory allocation.

Previous version 11.0.7000.967

Latest version: 12.1.2015.2015

If you've disabled or removed SEP, than it isn't the likely cause.

Has anything else changed on the machine(s)?

It doesn't seem to related to SEP however its hard to convince microsoft if you have AV installed ;) 

Can you plesae disable Tamper Protection and see if that helps.

If not take an output of netstat -ano and see if the server is running out of ports.

If that doesn't help we need take a Pefmon, Poolmon, Memory Dump from the srever to find the RCA of the issue.
 

Hello,

I agree with the above comment. Could you please PM me your Case #?

Let me look into the case.

Nothing changed on the server. all policies are intact no NTP installed only plain AV/AS only. Our Systems Administrator said that it never happened like this before, its just now that after the migration of SEP. No other softwares or other roles for this domain controllers. Basically, it happened now to 7 of our regional DCs. In past 2 weeks 7 of them causing this problems and since the last touch was done is SEP upgrade, the clients are suspecting SEP. But upon opening a case to Symantec Support, they said that it is not SEP which is kindly not clear as of of the moment. I already sent Symhelp logs but they see no cause of Symantec. Another support says that it would be better to send them also the Dump files for further investigation of the issue.

I thought that 12.1 RU2 has already fixed the memory leak problem but Symantec support Symhelp logs is not suffice to determine what really the cause of the issue. By the way, for the temporary resolution of the problem, the Admins restarted the servers in order to quickly restore the memory leak that causing the auhtentication problem of users. Admins are afraid that it would cause a bigger problem when this reaches their datacenters. All their systems, emails, firewalls and databases are all located there.

Now, as per the last task done by our admins they do server patching last sunday that causes all DC servers to restart one more time. We are waiting for the issue to rise again, waiting to see if the memory leak will still occurred and I hope it is not on Datacenter.

Hi Mithun,

I already PMed you my case id number for this issue.

Thanks!

Dear Joash,

the SymHelp report is not a universal tool for every kind of IT issue. It is not designed to investigate on suspected memory leaks. PoolMon is the tool for such things, plus memory dump analysis:

http://support.microsoft.com/kb/177415/en-us

According to my experience, SEP 12.1 might cause high non-paged memory usage due to the fact that, even if NTP is not installed, the Symantec network driver is still loaded.

Try to disable the Symantec network driver (it is not necessary when the NTP is not installed) and let us know if you still see the same issue:

SC config <DriverName> start= disabled

Where <DriverName> is:

You also need to distiguish a memory leak by "just" a high usage, it is not the same at all.

To be precise, the Support engineer sohuld see some memory usage report in the SymHelp report (yet not enough to spot a memory leak), but you can't see them by yourself without the SymHelp Viewer (Symantec internal only tool) hence, to deny that the issue is not caused by Symantec, you need to use Microsoft PoolMon tool and verify what process/driver is eating the memory.

Where is this SC config located? Ok, I think that's why also the Symantec Support asks for memory dump files....

SC is a Windows command...

ok let me not close this as we are still waiting for the problem to occur again.. right now, all DCs are normal and we are closely monitoring it.

Thanks for the suggestion and comments. Please do not close this issue yet, I really want to know the really cause of this issue.

Thank you.

When closing the thread is up to you...

ok. but even in SEP 12.1 RU3 it cannot resolve the problem , right?

Generally speaking, the upgrade is still suggested, however, a confirmation that it will help may come only once the investigation with the support on PoolMon outputs and memory dumps is completed.

still having the leak? Just curious if you resolved that in your environment

Cheers,

The memory leak didnt occur anymore when the server was restarted by our systems admin. I guess what resolves to that is restart....