Endpoint Protection

Please read on and see attachments (links below) for details............

I'm now thinking that SEP is actually responsible for most of our network slowdown issues.......... mostly because of what I see with GHOST interaction, and the fact SEP's firewall is alerting about things that can't possibly be happening! WE DO NOT HAVE IPV6 installed!
It's simply not here! but when we try to use a notebook, XP at that, as a ghost server, SEP chokes it down terribly, shows issues in the logs via Teredo hits (IPv6), and causes CPU and memory to spike, network throughput to tank.
Here goes - see below and attachments for many details and screenshots!!!

SEP has a problem and I think I can prove it.

SEP kills ghost processes and causes huge network traffic choking on some computers here, causing the ghosting process to choke, SMC shoots up in CPU and memory use.

We are trying to use notebooks as portable ghost servers, please read on, and see the attached documents for screenshots and more details.
------------------------
4 notebooks total:

Group A - 2 of the notebooks are HP 6730B with WWAN (Verizon over Agere modems supplied in the HP notebooks by HP)

Group B -
1 notebook is a HP 6730b with no WWAN support, no Agere modem, strictly Broadcom wired and Intel wireless support.
1 notebook is an HP 6320 with Broadcom wired and Intel wireless support.
---------------------------------------

The three 6730 notebooks are running the same Broadcom drivers (2 in groupA, 1 in groupB).

The top two have Cisco VPN software installed but NOT in use during the issues seen.
(I have actually tested with the VPN software removed and it made no difference.REMOVING the VPN software shows it's not involved)

All 4 notebooks are in the same domain, the same groups in the domain, and are running the same version of SEP in the same SEP group with the SAME policies applied. As far as SEP is concerned there are no differences between the 4 notebooks, none. All 4 are running SEP RU5 - the latest.
All 4 are Windows XP with SP3.

Ghost server is installed on all 4 notebooks, both group a and group b.
Same version, installed the same way.

The notebooks are used as portable ghost servers to prevent us from having to image computers out in the field across our WAN structure. They connect the notebook to the office network, start ghost server, and ghost their client's computers from there on the local network in that office.

Two of the notebooks work PERFECTLY, it takes only about 9 minutes to ghost a desktop computer using the GROUP B notebooks as ghost servers.

The notebooks in GROUP A have issues - SEP chokes the process, nearly killing it, causing a 9 minute ghost session to take an hour!

If we right-click and choose "disable SEP", the process works fine. However, all that is doing is disabling the NTP or firewall, as when you open SEP, the top two items are still running and enabled. Only the bottom, or NTP, is not running.
Ghosting from all takes only 9 or 10 minutes with SEP disabled, or the NTP disabled (the AV portion is still enabled in all cases)

On the two GROUP B notebooks there are no issues, speed of ghosting FROM them is good - there are no issues, it stays fast, the network shows a load, CPU is normal, SMC.EXE is not in the list using CPU or memory.

On the two GROUP A notebooks, it starts out quickly, then in 3 or 4 seconds, you can see the estimated time grow and the speed (meg per minute) drop quickly! SMC.EXE is hogging the CPU and memory!
When this happens, you can see the network traffic or load on those notebooks drop. The CPU load increases to 50% or better and the memory usage shoots up very fast. If you disable SEP (again, all this does is stop NTP) CPU used drops dramatically, memory use drops dramatically, and network load increases again (like expected as SEP is not choking the network)
If you ENABLE SEP again, CPU shoots up, memory shoots up, network load drops to a trickle and the ghost process slows to a crawl.
When SEP is enabled on the group A computers, their logs shows TEREDO traffic! There are numberous entries stating that IPv6 traffic is being logged. IPv6 isn't even INSTALLED on these ocmputers - they are XP machines! NOT vista.

SEP states IPv6 Teredo on the two notebooks in GRP A but not in group B! Yet it's the SAME SEP, same profiles/policies, etc.!

Woops - guess I can't attach these documents, limited to pretty small files size.......... please see these documents I've placed on my web server:

http://antique-engines.com/documents/ghost-Doc1.doc

and

http://antique-engines.com/documents/ghost-sep-interaction.doc



 

 

BTW - the GHOST SERVER, a Windows server 2003 machine with SEP fully enabled works great! No errors, no speed issues! So there's another success............

Get wireshark running on the problem machines to see the exact packets.
If there are any windows 7 machines on the network you will see ipv6 unless it is specifically disabled.

Also, have you played around with tamper protection as that is notoriously bad on notebooks with lots of vendor stuff installed.

Z

It's as if the SEP NTP can only handle so many packets at once, and the flood of Ghost packets chokes it, sort of like you have a 1" water pipe without SEP, and you turn on the water and it fills the 1" pipe to capacity, then turn on SEP and it is like putting a 1/4" reducer in the pipe, causing a drop in water flow, and a backlog upstream.
Or maybe better, your network is the Colorado river in the spring running with snow melt - and SEP is the DAM downstream - it slows the flow to a trickle, and the lake builds up behind it...............
That is how SEP is working on these computers - when SEP is off, the network traffic jumps to the top. When SEP is on, the network traffic slows to a trickle.

I will start to mess with the other protections and features, but I'm still baffled - it works fine on TWO of the notebooks, NOT fine on the other two.
The differences is that the two that do not work are equipped withe the chipsets for WWAN, the two that DO work are not equipped with the chipsets for WWAN (Verizon Wireless Internet, Wireless WAN)

Forgot to mention - these are Teredo IPv6 log entries not coming FROM OTHER machiens and being blocked, but coming FROM THE TEST NOTEBOOK! SEP states the packets are ORIGINATING from the test notebook with Windows XP!
How can an XP machine generate IPv6 packets? It can't, so SEP is seeing GHOST packets and mistaking them for IPv6! I checked the logs - they are packets FROM the notebook Ghost server destined for the client being imaged! Not from an outside source!

OK, I changed the policies for a test group.
I opened up the firewall, unchecked all the IPv6 rules, left everything wide open.
I removed ALL application and device control rules - either deleted or unchecked everything.
ALL devices are allowed, ALL applications can do anything on any drive, there's no checking anything, no blocking anything.
Everything, all sections, are as wide open as it gets - nothing blocked, nothing logged at all, all logging disabled, every piece of SEP has been told to do nothing, don't bother checking or looking, and don't write any logs, just let it happen.
AV is there, but that's not the issue - SEP's AV seems to be totally benign in this, totally blameless.
It's the NTP or anything that uses the NTP piece of SEP to operate as with NTP off, things work, with NTP on, things to not work.

And with everything either disabled or set to wide open, allow or don't check, if NTP is one, it chokes. If NTP is off, it works, even though NTP has no rules to check or block anything.
IT's simply the fact of it being active...............

Could you generate a custom sig in IPS to allow the ghost traffic to pass unfiltered?

I don't think this is "specific to" ghost. I think the ghost process is suffering because of an overall choking of traffic, and ghost happens to need the most throughput.

The ghost traffic passes, but is choked down.
The logs get numerous Teredo IPv6 entries, supposedly coming FROM the ghost server which is literally impossible. So SEP is totally reading some packets WRONG, and claiming them to be IPv6.
That's 1 problem right there. When you see only two IP addresses in the logs, one is the ghost client, the other the notebook acting as the ghost server, and the IPv6 Teredo rule is being triggered by the ghost server IP address sending TO the client IP address, and both are XP and it's literally impossible for IPv6 traffice to come from either, then SEP has a problem.
That's one of my points...................

Shadowspapa,

We saw something that sounds kinda similar.  The way I got around it was to add the IP addresses of my local network to the excluded hosts in the Intrusion Prevention policy.   Hopefully this will work for you too. 

Sorry, had to laugh a bit as if we do that, it points out another SEP flaw - if we exclude our own network, that means excluding our DCs/DNS servers and SEP will no longer do name resolution and in the logs you get only IP addresses, not site names! So we had to fully remove exclusions!
Put 'em in, halts name resolution, take them out, it again resolves names in the logs.................. SEP can't tell the difference!

 In IPS, you can exclude just one host.  Typically done when you have a scanning host for PCI compliance or the like.

But in regards to ghost traffic and what SEP thinks is IPv6, if the right signature is created, SEP will better understand and detect what the traffic is and wont be misclassified.  Thus you can better exclude the traffic.

However looking at the above ways of thought, seems like an IPS exclusion would be the simplest to try out.

Can you exclude everything except your DC's and DNS Server? Or is ghost running on one of them? I'm curious just to see if it has a direct impact on your issue.  I'm fortunate in that I only see this on a very small subnet (no DC's or DNS servers) and it only affects certain systems.   

These computers constantly change IP address............. they are NOTEBOOKS running GHOSTSERVER.EXE
So this means, they would have to add themselves as an exclusion once they knew their current IP address.
Remember, it's not really totally "blocking" meaning not allowing ANYTHING through, it's choking it terribly - and when that happens, we notice the teredo entries in the logs for that computer.
They would need to access the console, go into the group they are in, edit the IPS settings, check enable excluded hosts, click excluded hosts button, and add their own IP address each time they want to ghost something.

I have a funny feeling that it would take a Symantec engineer with one of these two machines in their hands, in their lab to figure this out. It's confusing, obscure, only impacts a handful of computers (AFAWK!!) and defies logic - and I've got all policies pretty much emptied - not blocking or checking anything at all............

More specifics:

With SEP not installed at all, clean-wipe used, using a notebook as a GHOSTSERVER gives me 690-716 meg/minute on the client screen - the ghost screen on the client being imaged.

WITH SEP, and with SEP enabled, it starts out pretty high - in the 600+ range, then you watch the rate drop by 10s and 20s until it gets to about 150 meg/minute or less with the remaining time INCREASING!  IT will hit about 4% then actually pause, the numbers stop moving........ it may start in again, but you will see several Teredo IPv6 alerts in the logs. I've had to change that rule to ALLOW Teredo, otherwise it will literally block GHOST. The only reason you can even ghost with NTP enabled at all is because I've changed the IPv6 rules to allow and not block. Otherwise, SEP WAS literally blocking ghosting completely!! Yes, SEP does block ghostserver transfers with the IPv6 rules in place so you have to change them to allow FROM block.

If you then disable SEP, right click, choose disable, and all it does is kill NTP, the AV part still running, only NTP disabled, the number cllimb quickly at the client hitting easily the 600s again while the remaining time quickly drops..

If you then re-enable SEP, the remaining time again climbs and the meg/minute drops quickly from several hundred down to under 200 - at times pausing.

So even with SEP rules set to allow and not block, it's still choking ghosting. With the Teredo IPv6 rule set to block and not allow, it halts ghosting at about 4% and won't allow it at all.

 So in other words, create a custom signature for the ghost traffic, and allow that, and help SEP understand that ghost traffic is not IPv6 traffic...

I still think the IPS custom signature is what needs to be fiddled with.

That's fine and great and all - but how the heck do you do such a thing?
IT's the firewall where the log entries are............
OTOH, with those firewall rules unchecked, there are no such log entries. There is in that case a huge slowdown with no log entries.
I see a lot of "advice" "simply create a xxxx rule" but that is like saying to someone "if you need to get to the west coast really fast, just rent a 747 and fly it there".
Ever flown a 747? I haven't - and I have no clue about IPS signatures NOR how to make it know what a ghost packet is.
Sorry, might as well suggest I reprogram the BIOS..............

But I still see the flaw in this as  - turn off the firewall rules and the log entries go away, but the slowdown still appears.
So how would some IPS signature, which I'm clueless about, solve a firewall issue?

http://seer.entsupport.symantec.com/docs/331096.htm

And

Wireshark or SNORT 
http://www.ibm.com/developerworks/web/library/wa-snort1/

What I'm thinking is that with the correct signature, it will be identified correctly as ghost traffic, and not be analyzed as something else or flagged as a false positive.

It already allows the ghost traffic............ at least when I uncheck the Teredo rule.
I don't want a band-aid, I should not have to create - spend all that time and energy and effort to create a rule when SEP should never ID ghost traffic as teredo IPv6 to begin with.
My point is *SEP is broken, not working correctly on SOME computers*, just SOME computers.
And if I uncheck the IPv6 firewall rules, all is fine as far as BLOCKING, but even with the rule removed, SEP is choking network traffic horribly causing GHOSTING to take an hour, instead of 9 minutes.

TWO issues - with the firewall in DEFAULT mode with the IPv6 rules enabled like SEP comes with, it blocks GHOST but only on SOME computers.
If un uncheck that rule, or check allow instead of block for that rule. GHOST gets through.

BUT, when ghost gets through, it causes HUGE slowdowns - throttles network traffic horribly.

Issue 1 - SEP's firewall incorrectly sees ghost traffic as Teredo IPv6
Issue 2 - with the firewall rule unchecked, it allows ghost traffic through but chokes it horribly.

This is WITH a rule at the TOP that says to ALLOW ALL TRAFFIC TO AND FROM the ghost EXEs. So I already have a rule, at the VERY top of the firewall that states to allow ALL traffic bound to and from ghostserver.exe to pass.
It does not.

This is the firewall doing this - NTP, not other parts - it's the firewall. When I disable the firewall, it lets everything through.
I don't see what IPS has to do with the firewall rules for one, and since I already HAVE a firewall rule stating let all things ghost through, why would anything else impact this? Do IPS signatures tell the FIREWALL to let things through? If so, what's the point of firewall rules?

The SEP firewall is the issue. It's not up to the customer to fix what is a Symantec issue - besides, the firewall already states to let through application traffic. Why does GHOST trigger the IPv6 rules? Tells me, SEP has internal issues recognizing traffic from their own application!

Before I spend hours attempting to figure out how to run wireshark, then hours making an IPS signature - I'd like to know that my 8 hours doing so will fix the FIREWALL, since the NTP is the issue - I don't see a connection between the firewall and the IPS signatures and need to justify my time spent troubleshooting a broken firewall with IPS signatures.........

 It's friday calm down.

My frustration comes I think from no one really understanding what this is doing, and NOT doing.
Both are Symantec applications.
I don't have access to the computers that are acting up since they are in heavy use by people and we have NO spares.

This is a firewall issue, and I see no relevence in other items........... I still have no explanation as to why I should spend hours loading and figuring out how to use "wireshark", then create IPS signatures, when it's the firewall causing the issue.
IPS signatures won't fix the fact that SEP sees ghost traffic as IPv6 traffic.
SEP's firewall ALSO keeps alerting us to DOS attacks, oversized packets, when that, too is a lie! There are no such DOS attacks going on.
SO, SEP's firewall can not properly recognize what's going on. It constantly, hourly false alerts on DOS attacks, tells me that IPv6 traffic is coming from a notebook with XP, FROM, not TO that notebook, it blocks GHOST via the TEREDO IPv6 rule, then when that rule is disabled, it causes a huge slowdown in the network.

If Symantec states I need to do something that will take hours to do as a troubleshooting item, that's one thing, but sorry, I have trouble connecting IPS signatures with firewall false alerts. No one to this point as explained WHY I should create signatures for IPS when the firewall is the issue and rules already exist.

MAYBE it's in part because I also know that we have computers here that take 10 minutes to open a single Excel spreadsheet - all since we started using SEP............................................................

Ghosting with SEP enabled, SMC uses over 50% processor........ and network utilization plummits.
SMC uses over 576,000K memory!
That's HUGE!!!!!!!!
Disable SEP and SMC.EXE drops to almost no processor use and normal memory.
Enable SEP and SMC.EXE skyrockets, sucking up almost all available memory, swapping to the paging file constantly, and processor is over 50%.

SEP has a real issue with SMC's memory and processor utilization, no way around it. With SEP's SMC using that much memory and processor, no wonder there's nothing left for other applications and things crawl to a near halt.

I've got the screenshots to prove it........

WOOOHOOO!!

More good news! For kicks, since no one seems to have the slightest clues at all as to why SEP chokes with ghost on SOME machines, I grabbed an old copy of wireshark a fellow here had from last fall, 1.01 or something like that.
Threw it on two test machines being used as GHOSTCAST servers.
On the one that works NORMALLY and from which an image can be pulled in 9 minutes and 6 seconds, all looked exactly the same with SEP enabled or SEP disabled, and the times were not different.
I'll call that a control computer -

On the OTHER computer, WOW, CONSTANT fragmented packets, I mean CONSTANT, tons of them!
AND, get this - NORMALLY with this computer as a server, SEP on means GHOSTING takes 30-45 minues, SEP off means GHOSTING takes 10-15 mintues.
Ah, but with WIRESHARK installed, GHOSTING TOOK 45 minutes PLUS! Regardless, SEP on or off!!!
WIRESHARK AND SEP seem to not play well together in the sandbox!!!!!!!!!!!!!!!!!!!!!!!!
In fact, the computer was so slow as to be nearly unusable and SEP was topped out on memory and CPU use!
I disabled SEP and it made no difference, but when wireshark was terminated, GHOSTING speed skyrocketed!!!
SO, it's some interaction, I'm guessing, of SEP and other stuff, and when SEP and wireshark run together on these machines, all you-know-what breaks loose, almost like a senatorial filibuster with infighting in the party..............

SEP has an inherant overhead
Wireshark has an inherant overhead
Anything that captures or analyzes every single packet has an overhead.

You need to approach this systematically and you are lucky you have some machines that work just fine and others that are broken.
That is the easiest troubleshooting as you just need to work out what is unique to the broken machines.

You mentioned WWAN cards. Try disabling those in the bios.
Ghoast is notorious for chewing up network bandwidth and used to completely kill an entire hub/switch back in the days when I used it.
What else is different on those machines?

Wow, someone read the details!

Anyway:

>>SEP has an inherant overhead
Wireshark has an inherant overhead
Anything that captures or analyzes every single packet has an overhead.<<

Yup - and that's a given - that's why I spoke with the network admin here and I'm going to used a DIFFERENT notebook to run wireshark and a HUB to connect things so I can remove Wireshark itself from the equation! Wireshark was killing ghost so GOOD testing wasn't possible.
SEP checks packets, and it seems to have large overhead in doing that, though on some machines, it's hardly noticable.

>>You need to approach this systematically and you are lucky you have some machines that work just fine and others that are broken.
That is the easiest troubleshooting as you just need to work out what is unique to the broken machines.<<

Yes and no - the machines that have the real trouble we have only TWO of them - and they are in the field in use by field staff and nearly impossible to get my hands on. We have NO spares............. so they have to consent to giving up their notebooks while I test.
My OWN, a different model, does exhibit some of the same syptoms - however, it's a different model, and does NOT have the WWAN hardware at all. IT's about 3 years old, the others are new this summer.

>>You mentioned WWAN cards. Try disabling those in the bios.<<

When the cable is plugged in, the software literally turns them off - they no longer show up in the device mangler. Unless you unplug the network cable, you can't tell they are there. Plug in the cable, they disappear, unplug the cable, they "turn on" again.
Now I know and you know, that can STILL be different - so it's worth a shot literally disabling in the BIOS so yes, that's worth a try...... but if that makes a difference, then why does MY notebook with NO WWAN hardware at all exhibit similar symptoms? Enable SEP, choke Ghost down, disable SEP, ghost speeds up.

>>Ghoast is notorious for chewing up network bandwidth and used to completely kill an entire hub/switch back in the days when I used it.
What else is different on those machines?<<

It is and isn't - using UDP packets it allows me to still work normally, doing other things on the notebook when SEP is disabled. It's only when SEP is enabled that GHOST chokes and the network slows to a crawl. IT's some interaction..............
On the notebooks where SEP works PERFECTLY with GHOST, you can run ghostserver and be ghosting a computer down the wire and stil use that notebook normally! IT's amazing.............