Video Screencast Help
Search Video Help Close Back
to help
New in the Rewards Catalog: Vouchers for "Symantec Technical Specialist" and "Symantec Certified Specialist" exams.

Server Network Adapter Teams Broken - Network Traffic Failing - NIC / Team / Adaptive Load Balancing

Updated: 23 Jun 2010 | 33 comments
Philonius's picture
0 0 Votes
Login to vote
Hello All;
 
This is an FYI that we have discovered on our SEP 11.0 installations. If you are running a Network Adapter Team on the Server, we have found that anywhere from immediately to up to two weeks after installation, that all network traffic begins failing. We have traced it back to the Network Team on the Server. Typically two Intel Nics in either an Intel Express Team or Adaptive Load Balancing configuration. If we break the team, and configure only one adapter on the server, then network traffic returns to normal. Our Specs, Windows Small Business Server 2003. We have witnessed it specifically on the S5000VSA server board. Typical Symptoms are suddenly no-one can log into server. Remote Desktop to server connects, disconnects, connects, disconnects. Hope this helps someone scratching their heads. On the first one, we thought that the Switch was failing.

Comments

mjgall's picture
15
Feb
2008
0 Votes 0
Login to vote

Yeah, we've experienced some of the same problems with our load-balanced terminal servers.  Some part of SEP client seems to cause incomplete connections on complex network configurations.

In our situation, we have 2 terminal servers load-balanced to a single ip address.  The terminal servers use a virtual loop-back adapter for direct-server return with our load balancer.  After a period of 6-14 hours of use, the server becomes unstable, unable to login even at the console, and preventing all new terminal services connections and logging a 1006 event in the system log:

"The terminal server received a large number of incomplete connections.  The server may be under an attack."

As soon as I removed SEP and re-installed SAV 10.1 the server ran stable again.

I only install the Anti-vrius and Anti-spyware client on the servers.  I've tried disabling and enabling the firewall and any threat protection options.  None of the changes improve the stability of the load balanced server with SEP installed.

Lyle Epstein's picture
05
Mar
2008
0 Votes 0
Login to vote

We are having the same exact issue. We killed our team on our Intel Nic's and it is now working. Is anyone at Symantec aware of this issue and is going to fix it?
Paul Murgatroyd's picture
05
Mar
2008
0 Votes 0
Login to vote

I dont see anything in our support database, has anyone here logged a call with support?
 
Are you running Network Threat Protection, or just AntiVirus/AntiSpyware?

Paul Murgatroyd
Principal Product Manager, Symantec Endpoint Protection
Endpoint twitter feed: http://twitter.com/symc_endpoint

Lyle Epstein's picture
05
Mar
2008
0 Votes 0
Login to vote

Paul, we called in this morning about it after finding it in this forum. We were on hold for about 30 mintues while the tech was researching it and we ended up hanging up. We have confirmed this issue as well. I have also emailed my Symantec sales rep about it as was told to downgrade to SAV 10.2 which is not a valid solution. For now, we are running without teaming on 10 servers until Symantec get's this addressed. As for hardware, we are running Windows 2003 servers with Intel network cards and are using the Intel teaming tool in the device manager for the hardware. Please let us know if you come up with a hotfix for this.
Paul Murgatroyd's picture
05
Mar
2008
0 Votes 0
Login to vote

did you get a call reference number at all?
 
if needed you can log a support call via mysupport.symantec.com
 
failing that, please PM me an email address and I'll "try" and log a call on your behalf (believe me, its not as easy as it sounds!)

Paul Murgatroyd
Principal Product Manager, Symantec Endpoint Protection
Endpoint twitter feed: http://twitter.com/symc_endpoint

Lyle Epstein's picture
05
Mar
2008
0 Votes 0
Login to vote

no, sorry, no reference number.
 
I PM'ed you my email address.
Paul Murgatroyd's picture
05
Mar
2008
0 Votes 0
Login to vote

got it, I just looked and you have a LOT of accounts with us... any chance you could also PM the contact ID you were using?
 
thanks!

Paul Murgatroyd
Principal Product Manager, Symantec Endpoint Protection
Endpoint twitter feed: http://twitter.com/symc_endpoint

Lyle Epstein's picture
05
Mar
2008
0 Votes 0
Login to vote

I PM'ed you
571411881's picture
19
Mar
2008
0 Votes 0
Login to vote

 

 

 

 

 

 

same problem - NIC Team causes unpredictable network behavior.

 

 

I have two HP NC7771 Gigabit NIC cards teamed up using HP Network Configuration Utility.. when I deleted the team, disabled the 2nd card, and reconfigured the TCP/IP setting to only use one NIC, the server seems to be functioning normally so far and not killing my internet connection as it did before. (it would totally kill internet access for everyone in the company within minutes after booting the server)

 

is Symantec aware of the problem with servers using teamed network adaptors? It is mission critical that I have load balancing on my server, and to sacrifice this in order to get SEP not to kill my Internet access is, frankly speaking, ridiculous. Please forward this to you developers.

 

Lyle Epstein's picture
19
Mar
2008
0 Votes 0
Login to vote

Are you using MR0, MR1, or MR2?
Paul Murgatroyd's picture
19
Mar
2008
0 Votes 0
Login to vote

does anyone on this thread have the MR2 beta or tried it in this situation?

Paul Murgatroyd
Principal Product Manager, Symantec Endpoint Protection
Endpoint twitter feed: http://twitter.com/symc_endpoint

571411881's picture
19
Mar
2008
0 Votes 0
Login to vote

it was MR1.. migrated from SAV10.1, and the install/migration went bad.. so I uninstall it., which wouldn't help with the problem until I removed the NIC Team...  just about to run Clean Wipe to see if that takes care of it, will update later when I get a chance to test it more..
Lyle Epstein's picture
19
Mar
2008
0 Votes 0
Login to vote

Paul- Where do we get the MR2 beta from?
Ongoing pain's picture
20
Mar
2008
0 Votes 0
Login to vote

Not sure if this will help.  Try editing the policy settings for Intrusion Prevention and checking the "enable excluded hosts" and adding the source IP(s) to the list of excluded hosts.

DoS attacks appear to be detected by Intrusion Prevention and the firewall exceptions don't affect the default (undocumented policies) in Intrusion Prevention.

pbogu's picture
20
Mar
2008
0 Votes 0
Login to vote
ernie's picture
18
Apr
2008
0 Votes 0
Login to vote

I just noticed this post.  I logged a case with symantec a couple months ago now.  Their backline engineering team is trying to reproduce this same issue using the same hardware which they just ordered.  I am using Intel Pro 1000 MT Server Adapters in an adaptive load balancing configuration.  The latest drivers are installed for the Nics as well as the latest version of Symantec (MR2).  If I disable the team and use one nic, traffic flows properly.  Also, I have only installed antivirus and antispyware protection, not the other options.
 
So yes, symantec knows and no, they do not have a solution.
dfosbenner's picture
17
Dec
2008
0 Votes 0
Login to vote

It's been 9 months since this thread was active - I was wondering if there are any updates on this issue?  I have Intel NICs that use load balancing on all our servers.  I'm having intermittent failures with LAN communications on servers with SEP Client MR3.  I'm not using anything other than AV/AS.  No firewall, etc.

Lyle Epstein's picture
17
Dec
2008
0 Votes 0
Login to vote

nope. We are still waiting for Symantec to get this fixed.... it's really a bummer.

dfosbenner's picture
17
Dec
2008
0 Votes 0
Login to vote

MR4 was released today to general (non-Platinum) support people like me.  The Symantec tech I was working with today (on a different issue) told me that MR4 will fix the issue with load balancing/teaming.  However, a quick search of the MR4 release notes made no mention of teaming or load balancing. 

 

What is the work around you all have been using for now?  No NIC teaming?  I can't afford to have servers dropping out, so unless we know MR4 is fixing this issue, I'm going to have to implement a workaround.

dfosbenner's picture
22
Dec
2008
0 Votes 0
Login to vote

I spoke with a Symantec tech this morning who told me specifically that MR4 fixes the issue with teaming.  I asked him why the release notes don't mention teaming or load balancing, and he said the issue wasn't specific to these, but had to do with the Teefer2 driver.  I don't feel completely reassured.  But if anyone wants to test and report back, have at it!

Serengeti's picture
23
Dec
2008
0 Votes 0
Login to vote

Teefer2 (SEP 11 MR3) and NIC teaming on HP e.g. DL 385 G2 do not get on - the network does not work at all! If  NTP is removed from the client installation all is OK.  Call open with Symantec and HP. HP have reproduced the issue. Ongoing investigation with HP and Symantec.

Lyle Epstein's picture
23
Dec
2008
0 Votes 0
Login to vote

Anyone have the link to MR4 since it is not out yet? I will test it with Intel Teaming and let everyone here know if it is fixed, but I need a link to the download to do this.

dfosbenner's picture
23
Dec
2008
0 Votes 0
Login to vote

It's been out since 12/17, but you'll need to enter your serial number on the Symantec site to get it.

Serengeti's picture
30
Dec
2008
0 Votes 0
Login to vote

David, I am trying to get a statement from Symantec on what teaming issues MR4 has fixed but the response is that there are no fixes for teaming in the MR4 release notes . . . Not sure why the tech you spoke to said there are fixes . . .  especially without anything to back that statement up . . 

Any further info appreciated

dfosbenner's picture
30
Dec
2008
0 Votes 0
Login to vote

I think the tech I spoke with was mistaken.  I asked why there was no mention of teaming or load balancing in the release notes, and he said that plus other issues were all resoved with improvements in the Teefer driver.  But when I went back and looked at the release notes, in the "Components in this release" section, there's no mention of Teefer2 in MR4 either.

 

Question for those with this issue - do you have any additional components installed on your servers besides antivirus/antispyware?  I am only using AV/AS, and didn't install NW Threat Protection (firewall, instrusion prevention) on my servers.

Lyle Epstein's picture
30
Dec
2008
0 Votes 0
Login to vote

I tried MR4 with Intel NIC teaming and I am still having issues. With teaming off, issues go away.... so I am wondering how it is fixed??

dfosbenner's picture
03
Jan
2009
0 Votes 0
Login to vote

Lyle,

 

I'd like to compare notes with you.  I have Intel PRO 1000 MT Dual Port Server adapters in a few servers.  I have the latest Intel v13.4 installed on one server, and an earlier v12.1 on a few others.  Thus far, I've had only one server fail to load with network connectivity, right after I installed MR3.  I immediately uninstalled SEP and haven't retried that one yet.  The other servers are running MR4, and have been running OK with teaming (adaptive load balancing).  All my servers are Dell PowerEdge.

 

-Dave

Lyle Epstein's picture
03
Jan
2009
0 Votes 0
Login to vote

I'm running ver. 11.xxx driver. I'll update my driver and then try it again.... good to hear you are working with teaming with v13.4 ..... Hopefully it is FIXED!

Lyle Epstein's picture
27
Jan
2009
0 Votes 0
Login to vote

Tonight I upgraded a client with SAV 10 to SEP MR4. They were waiting till I could confirm that teaming was fixed in SEP. The install went perfect, and yes, TEAMING is fixed now in MR4!  The NIC's in the server I installed it on were Intel Quad GIG NIC's, all teamed together.  So I guess we can put this issue to rest now. Thanks Symantec for fixing it!

Serengeti's picture
10
Feb
2009
0 Votes 0
Login to vote

we still have a teaming issue with the MR3 and MR4 firewall component.

The affected adapters are HP NC371i Multifunction Gigabit Server Adapters. Driver v. 4.4.15.0

When the firewall is installed, the NC371i adapters are no longer available in HP teaming.

So no change for our issue.

This has been reported to HP and Symantec, who are figuring out how they will work on this together.

 

Serengeti's picture
01
Sep
2009
0 Votes 0
Login to vote

Teefer2 and HP NIC Teaming failure

HP have identified the issue as an incompatibility betwwen the NCU and Teefer2. They have pointed the finger at SEP, as their findings show that Teefer2 changes the HP NICs identity in device Manager and the NCU can then no longer recognise them. Awaiting theoutcome of the meeting between HP and Symantec engineering.
Issue seen on HP ProLiant DL385 G2 with Server 2008 SP1

Serengeti's picture
16
Nov
2009
0 Votes 0
Login to vote

the conclusion form HP and

the conclusion form HP and Symantec is that the problem is with Microsoft due to the way in which adapters and protocols are bound together. There is hence no solution, leaving us with no guarantee that NTP will not break network connectivity in future.

Ole Stokie's picture
22
Dec
2009
0 Votes 0
Login to vote

Possible solution

We have had various seemingly unrelated problems. Investigation led to:-http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1370573
Supplemented by:-
 Post W2K SP2 network offloading has been passed to the NIC. It is enabled (by default). Not that it is stated, but when two NICs are in a team, two receive paths are in operation, so how do two (or more) NICs talk to each other in hardware to co-ordinate efforts?
This ‘Feature set’ can easily be disabled in the registry :-
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
EnableTCPChimney
Type: REG_DWORD
Values: 1 (enabled) 0 (disabled

Due to a change freeze we have not implemented yet but may be worth trying.