Endpoint Protection

I have recently had 2 instances of anomalous behavior on my network that I have been unable to find a solution to. The first instance involved an embedded rint server/controller on a KIP plotter. Very randomly, people could not connect to it with the proprietary software or ping it. Many or most other people could, though. But sometimes after 10 minutes, sometimes an hour or two, those who couldn't connect, would be able to, and others, who had no problem, were all of a sudden cut off. Nothing new was done to anything on the plotter, the plotter controller or the network in general. I checked everything: cabling, wall ports, switch ports, TCP-IP stacks on the workstations, AV status (we're using SEP), the plotter controller. In the end, it seemed to have been a hardware issue with the embedded server/plotter controller. Once we replaced that, everything seemed to go back to mnormal. But that's not the end of the story.

The next week, on Monday morning, several users said they were unable to access network shares or get to the internet. I tried the  usuall ipconfig /release & renew, reboots, etc... The common thread was that they could not ping the default gateway, in our simple, single router setup. Of course, most other people could. But then it started getting hinky, too, with people having their service return, while others who had been working were now cut off. The strange thing was that, even thought they couldn't piong the default gateway, they could ping other device on the LAN!  Again, the outages were random, changing and with no clear cause. I went through the whole checklist, but couldn't narrow down the problem. 

The next day, after talking with a Dell technician and making soem switch configuration changes, the entire network came down and we spent the next 12 hours rebuilding the entire switching config, as we have an iSCSI backend and two VM hosts. SO after getting everything put together with the Dell stamp of approval, things were quiet for a day. So, it was chalked up to switching config. Though it had run just fine with my config for over a year.

The next day, the limited outages to mapped drives began occurring again. I spent four hours with senior Dell technicians (we have Dell 5448 and 6224 switches) and senior level Juniper technician (the default gateway is a Netscreen 50), testing everything imaginable, checking logs, etc... While these random and changing outages were occuring, they could find NOTHING wrong with these pieces of equipment. As I said, while some people couldn't ping the default gateway, others were working just fine. Also, those who could not ping the gateway, COULD PING OTHER SERVERS/NETWORK DEVICES. Plus, for no apparent reason, their service would be restored.

Eventually, after about 4 hours, and because of no apparent thing we had done, network service resumed normally for everyone. It has remained that way for almost two weeks now. But yesterday, the strange issue with the plotter controller, which exhibited symptoms eerily similar to the default gateway, showed up again. As last time, and with the default gateway, it was limited, transitory and utterly random.

My question is, has anyone seen this behavior before as a result of some virus/rootkit/trojan-backdoor infection? My SEP doesn't show anything obvious, though I just had a user get a Trojan.Zeroaccess.B infection yesterday. I figure this strange stuff has to be either virus, or some rogue device. I can't think of any other explanation.

Any ideas or help would be greatly appreciated.

I would start with packet captures and review for any suspicious behavior, such as http GETs for exe or zip files. Illegal ICMP ECHO requests (icmp type = 8 and icmp code not equal to 0) or questionable ICMP types 13,15, or 17. Check for bot traffic as well.

Packets never lie.

Brian,

Thanks for the response!

Where would I be looking for this stuff? On the switch port of the firewall? I'm using wireshark, but I don't have alot of experience with doing packet capture.

You need to span a port on your switch so you can see all traffic for that subnet. Ideally, you will want to capture when the issue is happening so you get everything. But you can also capture when everything is normal so you have baseline of what your traffic should look like when there are no issues.

I am going to second Brian on that.  Data capture. 

Also, you said something in there...  ipconfig /release /renew

That suggests that you are running the entire operation on DHCP.  (minus servers and printers - I hope).

Do you have long or short lease times?

Static assignments (by MAC address) for any machines?

Have you received any error messages relating to IP conflicts?  Or is your scope large enough to handle all the machines on your metwork? 

- Sometimes, people bring in their own toys (Laptop, Tablet, etc.) which could be taking IP addresses from your scope and thus, what appears to be random blackouts, is really the DHCP server running out of addresses. 

Do you have a secured wireless network (that almost everyone has access to and knows that Keys/Passphrase)?

There had been some funny things with earlier 11.x versions of SEPM and DHCP.

I don't think those would apply here.

Have you tried restarting the service or server that is handling DHCP?

Well, thanks again for the response Brian and Jason. Please forgive me if I ask a million questions because I am rather novice with the packet capture stuff.  So here are my answers to Jason:

1. I am running DHCP. I have many more addresses in my pool than I have users/devices. I don't think running out of IPs is an issue.
2. I do have servers, switches, the default gateway, etc. on static IP.
3. I have a 7-day lease period.
4. No messages about IP conflict.
5. I do have a secured wireless network, and I have given the passphrase to our users.
6. I misspoke. I have SEPM v11.0.5002.333. Do you have any documentation on the DHCP issues?
7. I have not recently rebooted my DHCP server.

As to the packet capture.

1. I have done baselines.
2. I'm not sure what you mean by "span the port". What I have done, at the direction of the Dell tech, was to "mirror" the switchport to which the default gateway is connected to an unused port. I have a laptop with Wireshark attached to the mirror port, so I am getting captures. I'm not sure if I'm capturing just the traffic to that port, or everythingon the LAN. Is there any easy way to tell?
3. When you say check for bot traffic, what do you mean? I do have Barracuda Web Filter on my system, which I think should be scanning for that?
4. So I should be looking for ICMP 8, 13, 15, 17 and note their source and destination, I assume?
5. Will http: GET look just like that in a Wireshark capture?

Again, thanks for your help. I'm curious, as you both seem to have a bit more expereince and knowledge, what you gut feeling is on this?

Yes, mirroring is essentially the same thing. Ping a machine from another machine. If you see that traffic in wireshark it shouldbe set correctly. Just don't ping from the machine running wireshark or ping the wireshark machine. Needs to be 2 different machines.

You would need to block port 6666 and 6667 on your firewall. Barracuda is only looking at HTTP/S. IRC uses port 6666 and 6667

Set a display filter to check for traffic on port 6666 and 6667 tcp.port == 6666 or tcp.port==6667

Display filter for illegal ICMP ECHO requests icmp.type==8 && !icmp.code==0

Display filter for questionable icmp types icmp.type==13 || icmp.type==15 || icmp.type==17

Display filter for http GET for exe and zip files

http.request.method=="GET" && http.request.uri matches "\.(?i)(exe|zip)$"

Display filter for HTTP GET http.request.method=="GET"

These are just a few I may run but you can always mix and match.

Any peculiar connections to RUS/CHN IPs from PC inside your network? or PC that connect to other PC via random ports?

I'm not really network guy...i guess you can check it from the switch..

Or try run SEP support tool at 4-5 randoms PC inside your network....

Found a whitepaper from Microsoft, which also points to Symantec papers.

http://support.microsoft.com/kb/953615

However, they are stating, 11.0.4 as the culprit and issues resolved in 11.0.5

* * * * * * *

as good measure, either way, you should upgrade to the latest version of 11.x which is 11 RU7 MP2 I believe.  You can download the latest version from:

http://fileconnect.symantec.com/

* * * * * * * *

Have you checked the logs (symantec logs) on one of the machines who is experiencing issues?  Does it indicate anywhere that traffic from an address x.x.x.x has been blocked?