Endpoint Protection

Hi all,

I currently have deployed SEP 12.1 RU1 MP1 on a Virtual Windows Server 2008 R2 ent. Everything functions perfectly fine, however I receive notifications at completely random times (usually 1 or 2 times per day) of an unexpected server error, but I have no idea what it is.  The log is below.  Can someone please assist on fixing?  Thanks in advance:

looks ike agent information is not processed, can you open support ticket?

Hi,

" receive notifications at completely random times (usually 1 or 2 times per day) of an unexpected server error," --> in what sort of notification ?

I get them in two fomats.  Usually on the home screen under notifactions as well as email alert.  The email alert is very generic.  Below is the alert from email, but I have removed the site and hostname for security purposes. 

does the timestamp matches for such incidents?

Ah, it looks like a java web application error to me.

DOes reinstalling the SEPM and then reinstalling the Java Runtime engine an option to try ?

These types of errors are generally safe to ignore, but here is what is going on:

Clients upload logs to SEPM (they upload it to Apache via the Secars.dll module).
SEPM saves the data in the inbox file. In your case, this is a file in the inbox\agentinfo folder.
Next, asyncronisly, SEPM (the Tomcat module) will scan the directory and begin to process any new files that have arrived.
After processing the files (and storing the data in the database), it will delete them.

In your case, we are talking about AgentInfo files. These are also refered to as "OpState" files (short for Operation Status, or something like that). These are XML files the SEP clients upload that show the current state of the client. It contains things like, current revision the client is running, whether or not the client is infected, operating system, etc.

What commonly happens is the "data" portion of the XML will contain some special charater. Either some special symbol, or escape charater that is not encoded properly. This causes the XML data struture to be "broken". Generally the data is not neccissarily 'corrupt', it is just that the client failed to encode it properly. This is a SEP bug.

Often the 'bad' data comes from some registry key that the SEP client copied-and-pasted into the Opstate data (often the "BIOS" name and/or version found in the registry, but there can be other sources as well).

I personally did a lot of analyzing and isolating work to fix these kinds of issues in SEP 12.1 RU2. So hopefully if you upgrade to SEP 12.1 RU2 (which should be out soon) the issue will go away.

But there is no guarentee that SEP 12.1 RU2 fixed the exact same issue you found. To analyze this further the process is as follows:

Overview:
1) You need to enable the "troubleshooting" option in SEPM (see details below). This prevents SEPM from deleting inbox data after it's processed.
2) You wait until the issues occures again.
3) You find the file specified in the error log. Since you have the troubleshooting flag on from step 1, it won't be deleted and you can find it on the disk. (The extention will be .dat.bak)
4) Analyze the XML file (The Agentinfo or "OpState" file) for XML syntax error, usually starting for some unencoded escape code in the data portion of the XML.
5) If possible, see if the issue can be fixed directly on the client Example: Modify the BIOS version string in the registry (if that's the issue). Depending on the exact field that is causing the error, you may be able to change that data-value on the clients so it doesn't upload that data again.
6) Be sure to turn OFF the troubleshooting option once your done. If you leave the option on for weeks, it will eventually fill up your disk with already-processed logs.
7) Delete all *.bak files from your Inbox\AgentInfo folder (there will be a lot of them, you may want to use a DOS prompt: del *.bak).
8) Improve the quality of the SEP product by opening a Support case and explaining the issue along with the evidence of the defect. This is a SEP bug.

To enable the Troubleshooting option in SEPM do the following:
1) Open tomcat/etc/conf.properties file found in the SEPM install directory.
2) Find/add the following lines:

scm.log.loglevel=FINEST
scm.log.troubleshoot=/inbox/agentinfo

3) Save the conf.properties file.
4) Restart SEPM.

NOTE: Enabling debug mode greately increases the size of temp logs written on the disk. It's fine for a few weeks -- but you do want to eventually turn it off as it reduces overall performance and does use a lot of disk space for logging.

To disable debug mode:
1) Open tomcat/etc/conf.properties file found in the SEPM install directory.
2) Remove the scm.log.troubleshoot= line completely.
3) Put the scm.log.loglevel line back to INFO
4) Save the conf.properties file.
5) Stop SEPM.
6) Delete all the files under tomcat\logs
5) Start SEPM.
6) Delete all the *.bak files under inbox\AgentInfo

Just for completeness, the full troubleshooting line is:

scm.log.troubleshoot=/inbox/agentinfo;/inbox/enflog/client;/inbox/enflog/enforcer; \
/inbox/enflog/system;/inbox/enflog/traffic;/inbox/enforcerinfo;/inbox/learnedapp/apps; \
/inbox/learnedapp/computerapp;/inbox/log/behavior;/inbox/log/client;/inbox/log/lansensor; \
/inbox/log/packets;/inbox/log/security;/inbox/log/system;/inbox/log/tex/AvMan; \
/inbox/log/tex/legacy;/inbox/log/tex/LUMan;/inbox/log/traffic

with no breaks in it. But in your case, you only need the agentinfo line.

I hope that solves you issue. I would be interested in seeing you post a copy of one of your bad AgentInfo files.

Again, the issue is generally minor and can be ignored (although it means one of your clients is failing to update their status). But if need/want to fix it, that's how you do it.

Hi Ghent - Your response was excellent.  Thank you.

Now, a little more detail.  Based on what you said, I am wondering if I am getting those errors becuase of  what happened below (a previous discussion post).  Can you let me know if that would make sense and maybe how to fix it (or the solution above may fix it)?

Old Post:

I recently upgraded from SEP 12.1 RU1 to MP1 on a virtual Windows Server 2008 R2 Ent. ed.  I used the upgrade clietns feature in the SEPM console to deploy.  After the deployment, some agents work just fine, others appeared offline.  

Since these are production boxes, I was not able to reboot, so a system admin installed the MP1 agent ontop of the old ones, locally.  After, all agents reported in normally, receive updates, etc.  However, althought the are functioning perfectly fine, several have a download arrow and X next to their host name within SEPM, rather than the little computer with greeen dot (see below).  When you double click the engine, it's status is "installation failed, rolled back" but the current verision is the latest MP1 agent version and it works just fine.

Becuase they seem to be functioning fine, I am wondering how to get that X out of the dispaly, so it doesnt look  like there is an issue.  I have deleted the agent, and let report back in, but it did not work. Suggestions, ideas, comments? Can you manually update the database?

After typing the whole message below, I don't think this is directly related. To know more, you need to run the process above and identify which field is causing the issue -- that could give a clue why it happens and if it's possible related to your previous question. My bet is "no, it's not related" -- but let's be sure to let the evidence speak for itself.

Now, my thoughts on your previous topic:

First, SEP 12 installs using as "Side-by-side" techonolgy (or methodology). When you install a new version of SEP 12, like MP1, it doesn't install "on-top" of the old SEP 11.x or 5.x use to do. Instead, it installs along side of SEP. So technically they are both installed at the same time. (If you open the install directory, you will notice that version 12 will always have a folder with the build number. This allows side-by-side installs of different builds)

When you reboot the machine, the "active" SEP switches from the current version to the newly installed version. This provides you with protection for the entire process, so your systems is never "unprotected" during an update. It also opens the way to technology that let's you rollback to a known-good state in the case of in installation failure.

So if you are telling me that the Admins installed MP1 but did not reboot, I suspect that you're not really running MP1. I have had admins approach me with the idea of "Install now, reboot whenever we can" (which was several months away on production machines). Although in theory this should work, I advised against it just because we've never tested leaving a large amount of time between the install and reboot (that's not what it was designed for). Besides, if something goes wrong, you won't know if it was SEP or something else that happened in the mean time.

After the new SEP install becomes "active" (switch happens at reboot), it cleans up the old version.

Now, to your screenshot. The "Arrow" icon (with or without an "x") isn't really the SEP Client itself communicating with the server. This is the SEP Installer communicating with the server during the install process, before the client even runs or reboots and switches the "active" install. The installer may communicate again after you reboot and it attempts to switch the active SEP install. If there is a failure either before or after the reboot, you'll likely get an "x" on the icon.

So the clients are showing x's, it means there was a failure duing the installation of the new version. In your case it seems the rollback was succesful and the old client is working, and communicating just fine. But the new installation has likely failed. (If you're not going to try and address the issue immediently, it's probably OK to delete these "SEP Installer" entries, just make sure your SEP clients did indeed rollback successfully.)

If you're going to upgrade SEP, I recommend waiting until you are within at least a week of your reboot window. I also recommend against installing lots of updates along with SEP and then rebooting once -- especically if any of the updates are driver related. Of course, in production you always have to balance time limitations. But if you do install SEP ahead of your reboot window, if possible, reboot and let SEP go in before you start anything else. Then install and reboot again for everything else.

As one admin told me, "SEP is not a new version of [Microsoft] Notepad." - Kevin B.

That was a little long-winded, but I hope it was useful.

Ghent, many thanks for the reply back !

so in this case the old SEP will be automatically deleted to save disk space consumption ?

Well, I'm sure thats part of it. But regardless, who wants multiple versions of a complex product like SEP on the disk? You don't want some Windows component loading the old version of an AV driver.

And clarifycation about your "old post". Clients should show online, even when "AgentInfo" data is not processed correctly. However, some clients may continue to report out-dated status due to the AgentInfo failure. The problem with Agent Info failure in SEP 12.1 RU1 (Improved a bit in RU2) is that if one agent is sending bad optstate, it will cause a whole batch of agents updates to fail. So you may see seemingly "random" clients not updating things like AV Definitions.