Would like to add that the other administrator saw the same thing in the event log.

We are experiencing the same issue with SEP, MR3.  It is only happening to our Windows 2008 server (32 bit).  No DFS shares, but lots of standard Windows shares.  Server stopped responding to client requests about 5-10 minutes after definition update.  XP clients hung upon logon while their computers tried to map to the server shares.  Vista clients were able to log on and access the server shares without any problems.

https://forums.symantec.com/syment/board/message?board.id=endpoint_protection11&thread.id=16557&page=2

Look at my last post and try the netstat -an

Then post what you find.

Are you wanti the output of that command when the server has locked up?  I currently don't have auto-protect enabled, just have scheduled scans. 

I also experience the same thing as above, vista clients seem to be okay, and xp are not.

Do it before and after

turn it on for a minute then post results.

Alright, I could give the before, but i'm not sure i'm going to be able to do the after as I really don't want to lock up the file server during the day and it's really rather random when it happens and when it doesn't.  According to that one post you linked to "if there are many ports open on 9090 with a CLOSE_WAIT state. If so, this is a known issue due for fix in MR2" however we are running MR3.  You think this might still be an issue?

-John

I dont know if it is or not.

But if you dont check it out we may never know.

If anyone else is having this problem please let me know asap.  We are on MP2 MR2 and still see this issue occasionally.  Microsoft blames Symantec Endpoint 11, Symantec blames Microsoft and Dell (hardware) says its not them.  I do know the problem seems to go away when you remove Endpoint 11.  Anyone else experience this?
Thanks
Steve

Psihawk,

Can you post you netstat -an findings?

We have this same problem, yet again.  We've now tried each and every MR and MP up to and including MR3 over the life of this miserable product waiting for a fix.  MR3 continues to break DFS shares.  It brought down one of our production 2008 x64 servers this evening on day three since its fresh MR3 install.  We've done everything by the book and have tried toggling each and every scan option.  No luck.  It's frustrating to spend so much time giving a product now three chances to fix something so basic as protecting file shares.  MR3 was the breaking point for us.  They can't put quality into something so poorly designed and managed.  We've had enough of this product and have made the decision to switch vendors as soon as the evaluation work is done.  So sorry Symantec; you were a great vendor for many years. Hurry up MS Forefront Stirling.  I bet they get it right the first time.

SNAP!

Our 32bit file server is running Windows Server 2008.

On Thursday 25th Sept we upgraded our File Server from SEP 11 MR2 to SEP 11 MR3 and the next day we started experiencing lockups on our File Server.

It seems as more and more users connect to the File Server using network shares, these users start to experience lockups, after a while anything that anyone accesses on the File Server is unresponsive.

Disabling Auto-Protect and rebooting seemed to resolve the problem, but the reboot is necessary. Stopping the SEP services alone isn't enough as the lockups still continued to occur when the SEP services were disabled.

On Monday I tried to troubleshoot the problem, so I re-enabled Auto-Protect and things went down hill again...and fast. Exactly the same problems started to occur when more than a handful of users accessed the File Server. The server became so unresponsive that it wouldn't reboot and I had to hit the dreaded OFF button.

Yesterday I logged a call with Symantec, no news yet but the engineer wasn't surprised that I had this problem.

I have now uninstalled SEP completely from the server. Today the server is fine so I hope there isn't any lasting damage caused by SEP 11 MR3.

SEP on Windows Server 2008 has been a complete nightmare for us. Since the day we installed it we have had one problem or another which is related to bugs in the product. SEP on other platforms like 2003 and Vista are fine. Its only on 2008 that we have these problems...bugs.

I will point out that the file lockup issue wasn't a problem for us prior to upgrading to MR3. BUT...we had another issue with MR2 (Opening Excel and MDB data files over the network were slow as...) that resulted in us disabling Auto-Protect. So it is possible that this problem existed in MR2, we just were not aware of it.

Come on Symantec - FIX your product!

Open a command line and run the command netstat -an > capture_openports.txt

Look for port with CLOSE_WAIT state and tell us how many you have please.

Same thing for us.  File shares were slow (on any release) with a handful of users, especially those larger mdb files as you mention.  Add 100 or even just 50 users and forget about it; opening a shared file takes 10+ min.  Disabling Auto-Protect and stopping services doesn't help.  One must reboot, uninstall, and reboot.  Support of 2008 is indeed a nightmare.  I've read this same problem exists on 2003.  Our biggest 2003 issue was Endpoint consuming all available disk space via the \Windows\Temp directory on a few servers, particularly terminal servers.

JE_Empire

Was SEPM installed on the same server that you were having issues with?

or

Was it just a file server with only SEP on it?

What is your server specs?

KR,

The SEPMgr is a separate server.  Here is a breakdown of three machines for which one notices Endpoint breaking DFS the fastest and with the most user impact - a terminal server with 50+ users:

VM1) SEPMgr: W2003 x86

VM2) File Server: SEP MR3 with Antivirus/Antispam role only, W2008 x64, DFS file share, AD, DNS, and DHCP roles.

VM3) Terminal Server: SEP MR3 with Antivirus/Antispam role only, W2003 x86, GP login policies mount a few drives and My Doc redirect to file server.

Unfortunately, we can't really reproduce or test outside of the production environment; needs a load to break.  When SEP clamps down the share, no users are able to work on the ts.  No logs. No resource concerns.  Just buggy code in our opinion. Hardware is not likely any issue since these are VMs running on ESX3.5U2 which makes them rather hardware neutral.  These VMs have all resources they need.  Uninstall SEP on file server and everything is great.

Someone listed earlier that dell hardware and microsoft was blamed by Symantec, I'd like to say yes I have dell hardware but my fellow administrator is not using dell hardware--maybe they are two seperate problems, but I'd like to point out again as someone else has listed here I as well am running DFS on Server 2008 x64 and experiencing the file share lock ups.  I've had auto-protect disabled for several days now and there have been no issues. 

Hello All,

There are many things to try to resolved this issue so please call support so they can help you troubleshoot.

Many things to try by calling support?  Helen I don't mean to be rude, but at this point after multiple MR's and MP's this issue shouldn't even exist.  When I did call suppport, the response I got was it was a Microsoft issue and was sent a link about disabling something on the Network card.  

This didn't correct the issue.

I also have the same problem with Server 2008 X64 on dell hardware.  From what I have read it was supposed to have been resolved.   The only solution I have found is to completely uninstall SEP and go without any AV protection.  I have tried every workaround that i have come accross and still have no solution.  Even tried disabling SMB 2.0 to see if that would work.  Symantec needs to fix this ASAP and stop trying various workarounds.

Probably not the best idea to run Win Serv 2k8 outside of a testing environment let alone on a 32x system. Symantec also recommends only installing AV service on all servers. I wouldn't expect two virtually untested products to be stable enough in a professional environment...no matter what Symantec or anyone else for that matter advertises.

SEP is not a refined product but fortunately I have been running SEP 11 MR1 - MR3 with very minor problems in a 2k3 and XP environment. Main problem I've been experiencing is upgrading groups with new install packages deploy successfully to only 60% of group clients. Rest of clients need to be done manually which is a huge pain. I was hoping this would be fixed in MR3. I'm not even going to bother testing on my test 2k8 server until next release...at least that seems to be the consensus here.

Good luck and thanks for sharing.

I'm running SEP 11 MR3 on a couple 2008 64bit servers and on a 32bit windows 2003 server - same stuff happens on all of them  - when the servers get busy the file shares disappear for XP machines and will not come back until a reboot.  All running on IBM hardware.

I have opened a case with tech support and Symantec has provided absolutely no reponse - these guys need to set priorities and get thsi fixed

Is there any way that Symantec can take this issue and replicate it internally so that we do not have to be the guinea pigs?  I would love to use SEP on my 2008 file server, but I'm unable to continually freeze my server while testing possible fixes.  Surely a company with Symantec's resources can set up a test server and replicate the problem?  Please?

I am posting this message here at the request of another subscriber. This is my instance of the Windows 2008 Server/SEP11 MRS debacle.: We recently implemented a new Windows 2008 file server. We utilize DFS paths for mapping and redirecting access to files. eg. redirect mydocuments and application data to file server and also drive mapping to file shares via DFS paths. We installed the SEP 11.0.3001.2224 client and was only using the antivirus/antispam portion. Our end users (about 50 Windows XP SP2 with SEP 11.0.3001.2224 antivirus/antispam portion and about 45 Windows 2003 SP2 Terminal Server users) encountered daily disconnects from all network shares and about once/day our whole network would pretty much hang up with all users unable to function. Rebooting the Windows 2008 file server would get everyone back up and running - pretty much had to have everyone log off/on again to their workstation to get reconnected. This would happen between 2 and 4 times per day. Other symptoms included permission change issues, etc. You can imagine how my popularity rating in our organization plummeted while this kind of crap was going on! Out of desperation, I brought our Windows Server/Hardware consultant in and we tried many different things to rectify the problem for a few days but nothing made a difference. In "Googling" the problem, I got onto the SEP 11.0 trail. I have since uninstalled the SEP 11.0.3011.2224 client and voila "No more hanging/disconnection issues". We have been using this product successfully on Windows 2003 servers since MR2 came out, but with the new Windows 2008 server, this product has been a nightmare. Has anyone else encountered this issue and if so, suggestions on how to make this problem go away would be much appreciated. Thank you in advance to all who can help out!

I've been having the same problem over the last two months. We're running SEP 11. MR3 (Version 11.0.3001.2224) on a 32 bit Server 2008 file and print server.  Shares become unavailable to XP or Server 2003 computers while remaining available to Vista and Server 2008 computers.  Sometimes there is an error in SEP stating that "File System Auto-Protect is malfunctioning" and all options to disable SEP are grayed out.  Other times Auto-Protect appears to be functioning normally but the shares are still locked up.  Disabling SEP sometimes makes the shares come back for 5 mins or so but then they lock up again.  Have tried unsuccessfully to stop/restart the server service but it never stops completely.  The only way to 'fix' the problem is to reboot the server. 

Have rebuilt the server completely once.  Upgraded versions of SEP (wasn’t on MR3 when I first built it), reinstalled SEP, etc.  The problem only appears when the server is under a load (In other words -- In production) and usually shows up every couple of weeks to start, then weekly, then daily and then more than once a day.  Have seen this problem on a couple of other servers here too, so I know it isn’t just this one particular new Dell server.

Web searches have turned up this thread and one at Petri (http://www.petri.co.il/forums/showthread.php?t=25791&page=6).  Glad to know that others are having the same problem but discouraged that the only posted solution (MR3) has not solved the problem.

I too have had the dreaded SEP 11 MR2 and SEP 11 MR3 server lockups on file shares running Windows 2008 64-bit. This has created a real nightmare for me, and I was literally have 500 users practically give up on what I have otherwise run as a completely stable environment for 5 years without a single major outage. All of the sudden, once SEP was deployed, I was calling Microsoft, EMC, HP, and all of them pointed to a memory leak in the kernal mode "Symantec SEP Filter Driver". Once a load is put on the server for file sharing, whether it is SMB1, or SMB2 traffic, it doesn't matter, the SEP filter driver has a memory leak that seems to fill up non-paged pooled memory (kernel mode) and crashes the OS. Reboots fix it, but who wants to have to reboot twice a day just to run an anti-virus product? I spent hours and lost quite a bit of user confidence because I put my faith into the Symantec product, even after waiting a year to put it into production, after extensive internal testing in our organization. Microsoft advised us that literally thousands of customers are calling with this complaint, and Microsoft will not support their environment, including ours, by running SEP 11 in any fashion on Windows. They actually had me call Symantec Tech Support while on the line with the, and explain the memory leak and symptoms, at which point the Symantec tech support rep found an internal-only document referencing this issue and that it would be corrected in MR3. I was literally about to start testing MR3 on a semi-production system, however, it appears that you have all done that for me (I feel your pain!) and MR3 is still a completely unstable product. I must now go back to my management and explain how we have all worked with Symantec for over a year now, and this product is unusable and we will have to entertain alternative options. We had a virus this past week that Symantec 10.2 did not catch, and it has spread to 15-20 of our servers, and disrupted one server due to Symantec locking up when trying to scan the infected file today on a server. I have found Symantec products to be of good quality over the years, but the latest failure of SEP, and inability for Symantec to catch what many other products are able to catch now, is now going to be my pitch as the "dealbreaker" for moving away. As with all of the others in this forum, I bid a sad but personal fairwell to Symantec. I cannot risk my, or my department's reputation on the product any longer.

We share your sentiments and pain.  A SEP memory leak would make sense.  What a shame that customers and not the vendor have to troubleshoot and communicate this bug.  Thanks for the research and sharing with others.  It disappoints us most of all that Symantec isn’t participating in these forum posts.  Likewise, opening a ticket is a waste of time from what others state.  Are they really that clueless how serious this is?  It’s like they only care about the desktop (more license revenue) and not servers.  We gave up on SEP a while back and just dropped support renewal, yet wanted to share a new solution that we were forced to find to protect our 2008 servers.  Per the Windows Server Catalog  WindowsServerCatalog  of apps certified for W2008 x64, Kaspersky AV for Win Servers Enterprise Edition 6.0 is the only logo approved security product.  We are running Kaspersky in production with no problems; so the Russians are now building better products than Symantec?  Once Microsoft’s Forefront security matures and supports 2008 in all roles we’ll likely switch to them.  Who better to keep pace with changes in the OS.  Good luck.

>>Hurry up MS Forefront Stirling.  I bet they get it right the first time. <<

>>Once Microsoft’s Forefront security matures and supports 2008 in all roles we’ll likely switch to them.  Who better to keep pace with changes in the OS.<<

You ARE joking, right??? Did you know that MS's OWN security application was breaking word documents on MS server shares!?!

check in google, you'll see that the issue Symantec has with SEP also exist in MS's own AV and security scanning software! MS software deletes Word files!

What a joke! MS updates and patches are responsible for a great share of our problems with servers here. SharePoint, won't even go there - we've had multiple outside vendors come in and attempt to get it stable. IT's been reinstalled 4 times on 1 of our servers already. I describe Microsoft's products this way - messy at best and each time they attempt to get into security and antivirus, they fall on their collective faces.  We won't go there on a dare.  They can't keep up with their own OSs. Can't say how many times, various folks in MS blame the folks "across the aisle", the app people blame the OS people, the OS people blame the app people, service packs break apps, app patches screw up the OS.

We're even removing Vista and have decided we will bypass it totally, it's such a mess in an enterprise environment.

Sorry, I found your posts to be humerous!

Microsoft dealing with av and security is like hiring a fox to guard your henhouse.

with what my ISP tells me (of server2008 and SQL2008) and our experience here, I'm rather surprised to see so many have jumped all over server2008.

Not us, we don't use anything until MS has worked the kinks out and patched it at least once.

Our admin here said 2008 was too new and buggy and released too recently. Move it into production? Not here..........

(a list of agencies around here show a growing number of Linux servers instead! Running about 40% now)

ShadowsPapa wrote:

with what my ISP tells me (of server2008 and SQL2008) and our experience here, I'm rather surprised to see so many have jumped all over server2008.

Not us, we don't use anything until MS has worked the kinks out and patched it at least once.

Our admin here said 2008 was too new and buggy and released too recently. Move it into production? Not here..........

(a list of agencies around here show a growing number of Linux servers instead! Running about 40% now)

Your comments are unhelpful and they are off topic.

Please refrain from posting these messages unless you have useful information for people experiencing this problem.

The point of these forums are for people to share useful information and knowledge to help others.

Same issue here... 

Version 11.0.3001.2224 MR3 on Clean install of Windows Server 2008 running inside of VMWare ESX 3.5. We are using DFS if this plays into mix at all. 

Knocked out all file sharing on the Server 2008 and also locked up all workstations that had connectivity to the Server 2008 that was affected for over an hour today $$.  Workstations didn't even show a start button and couldn't ctrl-alt-del.  Not fun.

Had the issue a couple months ago on MR2 or pre-MR2 don't recall.  Opened a case and Symantec stated 2008 was fixed and supported even-though it isn't listed as a supported OS when you download MR3.  They stated it isn't listed because you can't install the System Center on it yet.

Please let us know when this major bug has been fixed!

What versions of AV are users using on Server 2008 if this doesn't work?

I have the same problem on Windows Server 2003 R2.  I have 12 servers that have just a/v and anti-spyware installed and they will randomly lose file share connectivity with their clients.  A restart of the server fixes the problem until it happens again and there are no errors in the event log.

I hope they fix this soon, I have already started researching other solutions.

Please post your netstat -an findings on your server.

Do you see many close_wait port 9090 connections?

No, I only see one and it isn't for port 9090.  Of course if this would have to be run when the problem is happening, that could be why I am not seeing it.  Because I have to get the servers back operational, remembering to run that command before I restart the server is not a priority.

wait a few hours and see if it gets more.

How long until the problem happens?

There is no pattern as to how long it takes to happen.  Could be a day, week or 2 weeks.  That is the frusturating part.  I only have it running on one server now and put the rest on 10.1.

We don't have the SAME issue, but it's the same with the issues we DO have.

Troubleshooting is TOUGH because there's no real pattern. Can't find anything in common except SEP.

The timing changes, the people impacted changes, shoot, a doc can cause issues for one person, and not 5 others, and they can work 3 days, then suddenly lose access to 5 files in a row, then are fine again. Can't troubleshoot while on the phone - the phone call could last a bloody week.

Issue would still occur with the symantec services disabled.  Odd.  Once we performed full uninstall of Symantec and rebooted server the issue went away completely.

After 1.5 weeks of no issues we decided to install 10.2.2 since I found a bug fix ID regarding this exact issue.  Fix ID stated it was resolved.  But this time around server crashed again within 40 minutes of having 10.2.2 installed and heavy user usage.

10.2.2 Fix ID:

Windows 2008 dropping network shares with AutoProtect enabled
Fix ID: 1296949

Server dropped network shares, would only uninstall Symantec partially and wouldn't shut down.

This is on a 32bit version of Windows Server 2008 Standard.

Symantec, please let us know when this is fixed.

I just wanted to chime in that we've been having this same problem too, albeit with SAV 10.2.2 (MR2) on Server 2008 x64 file server w/ DFS... running VIRTUAL! No Dell or anything.

So clearly this can't be a hardware issue.

I will be migrating back to Server 2003 x64 R2 and not renewing our contract.

Jumping in as well.  We run all Server 2008 x64 servers and I am now running them without Symantec.  It took me a while to figure out why my servers were dropping off the network left and right untill I realized that it was Symantec.  Installing Symantec 11 MR3 is an absolute nightmare on Server 2008.  Removing Symantec fixes the issue, so this is clearly a Symantec issue.

1) SEP should do an OS version check before install and warn that Symantec does not support Server 2008 yet.  This is a simple change and would save thousands of administrators unspeakable hours of trying to figure out why their servers are crashing.

2) Uninstalling Symantec 11 MR3 has a tendacy to crash the server and requires a hard reboot to get back up.  Then SEP still runs at statup and no longer shows up in the Add/Remove Programs list.  You have to re-install and uninstall to clean server up.

A tech called me saying that one customer resolved the issue by reinstalling the software... I don't know if this was for SAV or SEP. The tech said that the customer waited 1 week before reporting no problems and Symantec closed the issue after that. I told the tech I can't afford my server to lock up again and 1 week isn't long enough to determine if the issue was resolved because I've seen it go for almost 2 weeks before the problem cropped up.

This is extremely disappointing.  I went through this with 10.2.1 and my 2008x64 file server a while back.  Now that 10.2.2 was released I was just about to install it since the readme says the issues have been fixed.  Good thing I googled...  Now I'm very reluctant to install it...

So, anyone given this a whirl with MR4 yet to check for resolution?

It is amazing that no Symantec employees have replied to the last 3 pages of this thread. 

MR4 does the exact same thing.  The server locked up within an hour.  It happened with a Symantec technician on the phone. 

This is just sad.  Symantec is really dropping the ball with this product. 

Its not been an easy task to get AP working properly with SMB2 on 2008 machines thats for sure.  We currently have a couple of customers who are beta testing a new version of AP that seems to fix the issues with 2008 x64.  We thought we could get it fixed in time for MR4, but sadly due to the lack of customer calls and physical memory dumps, along with some issues with the code it wasn't possible.

Those posting about MR4 MP1 are correct, the associated defects are indeed targetted for that release, but as others have said, it takes a long time to reproduce the issues reliably, then we need to analyse dump files and rewrite code in order to try and fix the issue.  With something as complicated as AP its a long time consuming process, which can sometimes lead you to discover further issues (not neccessarily with our product either) which need to be fixed or worked around.

Believe me, we want to fix the issue as much as you want us to, but we want it fixed properly first time.

Paul,

For those of us using SEP MR4 with no other options, how do we protect our 2008 servers?  Does Symantec have an official workaround for the issue?  I need A/V on my servers.  At this point Symantec has to provide a solution, or I have to find a new product.

Your input will be much appreciated. 

Thank you Paul for the update.  It is nice to see an official acknowledgement of the problem and a possible time when it might be resolved.  I understand how difficult it is to reproduce the problem.  That is what is scary for us because it is all but impossible for us to test our configuration for it.

Thanks for the clarification, Paul. The Release Notes for MR4 includes the following, which had me all excited until I read reports in this thread that it wasn't fixed in MR4:

Windows 2008 dropping network shares with AutoProtect enabled
Fix ID: 1290133
Symptoms: Network shares become unresponsive after installing Symantec Endpoint Protection MR2 with AutoProtect enabled on a Windows 2008 server.
Solution: Modified Auto-Protect to address the problem.

Is that a different problem, or an error in the Release Notes?

The defect fixed a very specific issue with SEP on 2008.  We had an automated test created as part of the issue reproduction that we used to ensure the defect was fixed - it was, the test passed and fileshares did not crash.  That doesn't mean that this issue and another issue with the same result are the same problem - far from it.  Thats one reason why we need as many people as possible to log calls with support and provide crash dumps when problems like this are encountered.  We are certainly dealing with more than one issue on 2008 file shares and the more information we can get from as many different sources as possible, the more we can ensure the product works in all circumstances.

To those asking about workarounds, the only real workaround at the moment is to disable AutoProtect on fileservers.

We agree thats not good, and as I've said, we are working to fix the issue once and for all with MR4 MP1.  In the meantime I would reccommend running scheduled scans every couple of days (you can run them out of hours) and ensuring that network scanning is enabled on the client machines as that should pick up anything lurking.

Paul, can we assume that the ultimate fix will also be ported to 10.2.2?  This is what I'm running currently and I have run into the exact same problem on my win2008x64 file server...  I am currently running periodic scans via the network from another server but as you know this is not really a workable solution.  Please let me know!

thanks,

Wes

Wes, Yes, once the version of AP is available that fixes the issue, it will be put into both products at the earliest MR or MP

This issue also occurs on Windows Storage Server 2003 R2 64-bit edition.  Within two days of installing SEP MR3 AutoProtect reported an issue and employees could not access the shares on the server.  A reboot solved the issue, but it reoccurred one day later.  We've removed SEP from this server and have had no issues since.

Has any progress been made on this? We are having all kinds of problems on domain controllers as well as file servers running SEP MR2 and MR 4 on Windows Server 2008 64-bit.

Also, will the minor release be available via the live update option, or will we have to do a complete reinstall of the server and clients.

So far my server 2k3 with latest updates and MR3 doesn't last a full night without a lock up. An uninstall fixes this, meaning the server works fine without Endpoint. Happens with just the management console installed and not monitoring any clients. No errors in event logs. Just a huge time gap of when it locks till when I hold in the power button to reboot it. Pretty much went through all the suggestions and even ran just the management without any AV installed. The last events before the lock are: Application Event > secars started.  System Event > The LiveUpdate service entered the stopped state. This is actually after a full unistall and re-install. Going to try tonight just having the program loaded not doing any schedule tasks such as scans or liveupdates.

We have 2 2003 servers with MR2 clients and SEP console on one of them. This server with management console is also a file server. The other one is Exchange. We are not having any lockup issues with them.

Hmm. Server is a file server but has been retired from public use so it doesn't get much use besides me messing with it. I'd say it may be the server but it runs fine with symantec 10. Basic default install with no changes to anything still causes lock up. It even locked within the hour of a install at one point but it seems to be an overnight thing. Backups run overnight so now going to look  into that as well  :/

Just adding my situation to the list in hopes of resolving things.

Setup:

Server 2008 x32 Setup as a domain controller.

Serving 52 Users

With Roaming Profiles and Folder redirection (Desktop and My Documents) for each user

Serving 15 Shared Folders (very light use)

We have tried both MR2 and MR4 (MR4 is a NIGHTMARE..Will explain/vent later)

With both versions the install was tried with both managed and unmanaged clients (SEPM is on a diff W2k3 machine)

Both resulted in random late night server lockups with NO event being logged. Both also resulted in every user experiancing a seemingly random set of errors. The most common of which was login/logout times exceeding 5 minutes (if they could login at all many would lock up at that point). Other errors were random disconection from the server, complete network stack crash(client side), extra orinarily slow responce times with any file located on a share (from slow opening / saving, to excele taking 30seconds to register a keystroke)

All the above errors vanish as soon as SEP is removed.(Forceably i might add, no joy in tryign to run the uninstaller!)

Now here is the MR4 nightmare (not Server 2008 related but i must vent!).

We rolled out MR4 building wide as the clients have not had a version upgrade since 11.0.1000. The upgrade was pushed out via SEPM to all users after we had performed succesful upgrades on test machines manually (thats the key and i should have known better). The issue happened the minute the update went out. Here is the chain of events as best as we can peice them together:

1. Push out upgrade and inform user that update is coming and offer to postpone (which did nothing BTW)
2. Copy the install files to THE LOCAL USERS APP DATA FOLDER!!!!!!!!!!! (note this is not by choice this is the default behavior it seems)
3. Begin install (Again regardless of wether or not the user optd to delay it)
4. During install process disconect from the network to reinstall the teefer2 driver for the firewall.
5. Clients suffer the following  (randomly and ever changing)
   1. Network permanently killed
   2. Windows refuses to boot at all stopping at a completly blank desktop with no errors
   3. Error mesages about incorrect version being called that will not go away (then windows crashes)
   4. Random 1-4 second disconects from the network
   5. Logins taking >45 minuts
   6. Unable to logoff or restart PC
   7. -or-install seems to go fine then dies within 24hours with one of the above

The errror is partially from the installer losing connection to where the install files are stored (remember roaming profiles?) I say partially because thats the best i can figure out from watching it do it on a test machine a few days later and from the error msgs.

So i thought that i could just spend the WHOLE NIGHT uninstalling/reinstalling from a local login with the setup on a local drive, seems I am 0 for 2 at this point. I CANNOT UNINSTALL THE CORRUPT VERSION... nor can i install over it.

System restore? Kinda works but then i must manually stop all symantec services in safe mode before i can do anything (see error #3)

So I get ahold of cleanwipe and tear out SEP COMPLETLY. Install the original SEP and swear to never..EVER do that again.

As an interseting aside..cleanwipe v3 works wonderfully on Server 2008!

Curious if anyone else running roaming profiles has seen this issue?

P.S. Forgot to mention that all the client machines of of different make / model (Dell/custom/hp/etc..) and all but 4 were running WinXP SP3 (the others were SP2)

Well Since last friday I've installed MR4 and changed my update schedule to weekly schedule instead of the check every 4 hours. Also unchecked the schedule scans. This morning was happy that the server was still running over the weekend. I went and changed my Update schedule back to the 4 hour and rechecked the scans and as typing this my server has locked!. After a hard reboot now I get jvm errors when trying to run SEPM. Well off to search for more answers in the MR4 posts. Good Luck!

Microsoft SBS Standard 2008 64bit - Windows XP and Vista clients losing connectivity to Windows Server 2008 64-bit file shares . Running Symantec Endpoint Protection 11.0.4000 MR4 (same issue in MR3)

Problems accessing file shares on Windows Server 2008 running Symantec Endpoint Protection

Question/Issue:

Why are Windows XP and Vista clients losing connectivity to Windows Server 2008 64-bit file shares and DFS shares?

Symptoms:

Windows XP and Vista clients lose connectivity to Windows Server 2008 64-bit file shares and DFS shares after a period of time when the server is under load during production hours. The Windows Server 2008 itself does appear to be working and does not show any signs of a deadlock. This issue was reported on Symantec Endpoint Protection MR3 and MR4.

Solution:

Symantec is aware of and investigating this issue. This document will be updated when new information or a solution is available.

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2008100113145148?Open&seg=ent

just adding my bit to this thread.

We are using Windows 2008 x64 as a file server and ever since the install of SEP 11 MR3 we also experience shares lock ups.  An uninstall of SEP seems to have helped the issue somewhat, although we find the problem is still there even after the uninstall!!  A reboot unlocks the shares, but that's hardly a solution.

One thing I noticed when the shares locked up was that there were numerous active conncections in the CLOSE_WAIT state when I did a 'netstat -na'.  600+ of them.

Now we have about 100 users that use the file server, but each user will display several connection in the CLOSE_WAIT state using ports from 1000-5000 range.  Server side is using 445 for SMB.  Each connected user can have many many CLOSE_WAIT's...

TCP    192.x.x.x:445       192.x.x.x:3922     CLOSE_WAIT
TCP    192.x.x.x:445       192.x.x.x:3933     CLOSE_WAIT
TCP    192.x.x.x:445       192.x.x.x:3951     CLOSE_WAIT
TCP    192.x.x.x:445       192.x.x.x:4251     ESTABLISHED
TCP    192.x.x.x:445       192.x.x.x:1061     CLOSE_WAIT

I suspect that is what's causing the share lock ups, since port 445 (for file sharing) is all jammed up by these connections stuck in CLOSE_WAIT.

Perhaps we didn't completely erradicate SEP from our server and some remnant of it is still hanging around.  

Anyone also see all these CLOSE_WAIT state on their server when their shares locks up?  Solution?

peace

Paul Murgatroyd wrote:
The defect fixed a very specific issue with SEP on 2008.

Paul, we have a WS2008 x64 DC/File/Print server that was previously unusable due to dropping shares (SEP11 MR2 MP1, IIRC), so we took it out of production. With MR4, we put it back on the network to see if the "specific issue" you describe was our "specific issue", because we REALLY need to put this machine in production. So far, we've not had the problem with MR4, so maybe this is our "specific issue"...

...but...

...the stakes are high, because this will be the ONLY DC/File/Print server at a branch office with no on-site IT. So, before we really commit to this machine, we'd like more information about the "specific issue". If there are System Monitor metrics or something measurable we can look at over time to predict if we're going to run into trouble, please elaborate.

TIA

We think we might have cracked this one.

Our engineers have been working long and hard (along with Microsoft) to try and get to the bottom of this problem.  At the moment we have code that has been running in our test environment for several days with a medium level of network stress (this test has always failed before) without issue.  We are currently rolling the code into a binary and then need to look at timescales.

At this stage its too early to comment on whether it will make MR4 MP1 or not, but we are trying to do so.  If we don't make it for MP1 we will be looking at how we can provide the file as a hotfix as soon as possible.

Jeff, its going to be difficult to tell if your problem is the exact same one, since we would need to see memory dumps from when your server became unresponsive to compare to the ones in the defect.

Paul Murgatroyd wrote:

We think we might have cracked this one.

 

Great news!

Paul Murgatroyd wrote:

 

Jeff, its going to be difficult to tell if your problem is the exact same one, since we would need to see memory dumps from when your server became unresponsive to compare to the ones in the defect.

Empirically, we think the MR4 fix might just be the one affecting us. Our test WS2008 DC/file server where SMB was hanging with older versions has been running without problems so far. I doubt it will be in production before the next MP, so I'm crossing my fingers that the other fix(es) are included in MP1. Then we can proceed with more confidence.

Paul, any update on this hotfix?

Running SEP MR4 on Windows 2003 Standard Edition Server, SP2 and having the same issue. Windows 2008 servers behaving the same. We installed SAV 10.2.2 on Windows 2008 and at the end I had to disable SmartScanning and Scanning the shared folders.

I'm reading about this fix for MR4 for SEP, just a quick question, where can I get it.   I'll take the hot fix if there is one, rather than waiting.  Our network was brought down to it's knees last night and everything listed about this, with the exception of us looking at the ports using netstat -na, identifies exactly what happened.  There was no warning, simply everything in the network came to a slow crawl, then stopped. I can't have this happen again.  We'll disable/uninstall if needed to prevent it, but it simply can't happen again.  Where can I find this fix you are talking about?   I've done the live update, I've went out the the download center looking for it, and nothing.  If you can help out, it would be a great help.   Thanks to all that have been putting an effort into this event also.  

Let me know.

What isn't funny is that you call Symantec tech support and they still tell you that this was been fixed in MR2. For the moment I disabled File AutoProtect and increased my scheduled scanning to every night.

Correcting my message from yesterday...we've been running the test DC/server I mentionned for a month on MR4 with no problems (albeit with a few restarts once we were certain the problem was solved).

This morning, SMB bit the dust. So the problem that was fixed was either not our problem, or not our only problem. Glad we didn't put it into full production.

I guess in AV terms, one would say our previous results were "false negatives".

Paul, don't know if this matters to anyone at Symantec now that a solution has been found, but I had the IT director pull the data plug on the WS2008/SEP11MR4 test server, but leave it running. It's in the same state as it was when he pulled the plug. Except that I've logged on to it with an admin account and run NETSTAT -AN for the heckuvit (nothing on 9090, 120-ish CLOSE_WAITs on 445, if that matters any more--or ever did!)

I don't have any immediate need to do anything with it because it's not in production. I'll leave it as is through at least Monday. If you need anything from it, please let me know.

Thanks!

Paul,

I have an open ticket number 312.102.518 and I submitted to Symantec the log files I ran on my Windows 2003 server and one of the WinXP clients. I rebooted twice this morning. One before tweaking the policies and 30 minutes after disabling File System Auto-Protect.

I set LiveUpdate to 5.00AM everyday and every night scheduled scan at 1.00 AM.

After second reboot it's hard to believe that disabling Auto-Protect is going to work.

Conestoga Rovers wrote:

Paul,

I have an open ticket number 312.102.518 and I submitted to Symantec the log files I ran on my Windows 2003 server and one of the WinXP clients. I rebooted twice this morning. One before tweaking the policies and 30 minutes after disabling File System Auto-Protect.

I set LiveUpdate to 5.00AM everyday and every night scheduled scan at 1.00 AM.

After second reboot it's hard to believe that disabling Auto-Protect is going to work.

Try adding this reg setting and see if it fixes it. I did it on my 2003 DC that has a few shares and it works well with MR3. I also added it to my other servers as well.

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters create a DWORD called TcpTimedWaitDelay=40

Thanks Knottyropes. You are still using this fix without problems?

And Symantec couldn't see your post from September 24, 2008?

He there

Just to know: We had the same promblems with multiple costumers - only solution; leave the Servers unprotected... And that can't be a solution at all... We had to stop now all EP11 rollouts; over 300 licenses are affected...

We hope for a soon solution...

Greetings

Mike B. Wehrli

SERCON AG

Conestoga Rovers wrote:

Thanks Knottyropes. You are still using this fix without problems?

And Symantec couldn't see your post from September 24, 2008?

Yes on all of my 2003 x32 servers. No lock ups with shares now.

I had a domain controller running fine for 175 days before that.

https://forums.symantec.com/syment/board/message?board.id=endpoint_protection11&thread.id=16557&view=by_date_ascending&page=2

I'm not having this problem on my W2K3 servers, so I'm not able to really contribute much...I am curious for those having this issue if disabling Digitally sign communications (if server agrees) in GP helps or not.  Particularly with W2K8 Server.

I've heard of a cuople of sites with XP workstations having connection problems to W2K8 servers (while Vista clients work fine) when this is left enabled (it's enabled by default).  I recall at least one poster in this thread with the same type of network, so I thought I'd toss it out there.

The setting is under Computer Config -> Windows Settings -> Security Settings -> Security Options -> Microsoft network client: Digitally sign communicaitons (if server agrees)

Just a thought.

Hi Paul, any news on the fix?  We are all ears!!!

thanks,

Wes

Hi Paul

Do you now when the MP1 will be released ?

We have several 2008 servers and experience the same problem. 

We have disabled Auto-Protect and runs a daily scan of our servers.

No fully good solution, but it works for now.

IMO, this issue is so serious that if MR4 MP1 isn't imminent, but a hotfix can be prepared, it should be made available as a separate download. Likewise, if MR4 MP1 is out the door before the fix can be included, it should be made available as a separate download.

We've removed this off one of our file servers.  It has crashed several times in the past couple days and we're trying to remove the egg off our face for still having these issues.   I agree with Jeff from the  prior post, that this needs to be offered as a hot fix and be downloadable NOW.  If your not sure this is the problem, then find out what it is and fix it.  We're currently looking for other types of AV if this is not going to be addressed, granted some of our options are not as robust, but at least those options would work.  Our largest file server is currently unprotected, not by choice, but simply because we can't have the server crash in the middle of the day.   It's not Symantec that's taking the hit here, it's the ones who run the boxes that are getting hit.  How can we stress this enough to Symantec to do more to address this issue, and get a valid working fix out here.  I'm still hopeful that Symantec can fix this QUICKLY, but I'm not going to hold my breath.

Call, open a case and insist it get attention.

They don't seem to watch what happens here to a great degree.

This is more of a gripe forum and peer-to-peer support, not Symantec technical support.

At least not like the OLD DAYS when I was a Symantec Section Leader, and then when they moved from CIS to Internet, a SSV - then, Symantec regulars, their online technical support people were involved directly and replied to what we could not or did not, or added to the discussion.

Now it's hit and miss - they may see a thread, they may not......

I miss the good old days!

I'd be thrilled to be on "the team" that was Symantec's Online Technical Support.

I still have one of their hats and a couple of t-shirts from those days!

It's actually more cost-effective to have online support than telephone support. It worked, it was quick and a number of folks were paying attention - including us SSVs and their regular staff.

Forums are CHEAP to operate, nearly FREE.

And staff can specialize. If someone knows more about policies than installation, they take those questions first.

I knew more about virus workings, specifics of how things infected, how they worked, and the corporate console inner-workings, so I handled a lot of that.

Alas................

Anyway, to get attention, CALL THEM.

We SEP-on-x64-server admins are too busy holding off the angry mob and rebooting our servers every hour to make a phone call!:smileymad:

Oh, how I understand (re: the WORD FILE slowdown with sEP's app control enabled)

I understand...... fully.

Good luck!

64 bit is becoming VERY common, even though we don't have any, and really no big plans to do so real soon. (State budget cuts of 6.5% might have a little to do with that)

I don't know if anybody else noticed it, but when I get someone from India working on my case I feel more responsibility from this side versus someone from the states. The Indian guy is really trying to solve my issue, he is working with me via webex and looking deep into the problem. I am talking only from my experience with Symantec Tech Support.

People in India have pride and are tickled to have ANY job.

folks in the states take it for granted, it's their right.

There's also a lack of pride.

My co-worker here has gotten 2 Cisco support people disciplined and sent back to customer service training classes in the last couple months! LOL - NO KIDDING!

I logged a support call with Symantec towards the end of September 2008 about this issue.

For the first month my engineer was helpful and gave me regular updates about their testing, but I have had nothing since.

My engineer was only interested if I was prepared to re-install the software, and collect dumps after my server crashed.  He didnt give a flying fart that it was a production server and an outtage would cause hundreds of people to lose their work and stop them working for 30 minutes while the **** was hitting the fan.

I love Symantec.

Hi,

FYI that I initiated another support call today with Symantec support as I have multiple clients who have this issue. I wanted to know the status of the MR4 MP1 patch that has been discussed here.

For those asking about a hot fix, he said there are no plans that he has heard about an early hotfix release...

The tech is telling me that the update is being tested now, and is slated for release end of February...

I explained the severity of this, clients leaving Symantec, etc. He totally understood and said that is why there is a delay with the release. They need to get this right and don't want to screw it up.

If there are still problems after this release, I will likely look for other AV vendors. I don't have the time or resources to deal with this kind of crap.

In the meantime, he recommended these workarounds. Hope it helps someone.

Disable the Rescan cache when new definitions load in the Auto-protect settings on the Server:

From the SEPM:

Edit the Antivirus and Antispyware policy associated with the Group containing the Windows 2008 server

Open the File System Auto-protect settings

Open the Advanced tab

Click the File Cache… button and uncheck the Rescan cache when new definitions load option

Push out the policy to the SEP clients

From an Unmanaged SEP Client on the Windows 2008 server:

Open the SEP GUI

Open the Antivirus and Antispyware Settings

Open the File System Auto-Protect tab and click the Advanced Button

Uncheck the Rescan the cache when new definitions load option

Another Workaround:

Setup Auto-protect on the Local Windows 2008 SEP client to exclude scanning the shared folder(s) and/or DFS root (has been confirmed to work for one customer). There are security implications to consider when implementing this suggestion. Ensure all end users OS SEP clients have network scanning enabled in Auto Protect and also ensure Autoruns is disabled in the customer environment as there are numerous threats that use the Autorun feature as an attack vector. See http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2008032111570648 for further information.

Hey Knottyropes, I tried your registry tweak and 33 hours later I had to reboot my W2K3 server again.

I just went back to the workaround suggested by Symantec, disabling the File System Auto=Protect and increasing the scheduled scanning.

Try adding this reg setting and see if it fixes it. I did it on my 2003 DC that has a few shares and it works well with MR3. I also added it to my other servers as well.

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters create a DWORD called TcpTimedWaitDelay=40