Not sure if I am missing somethign regarding the logging, but ever since I made a change from the default settings, my log keep growing.  It used to be about 40GB, now it is 450GB in about a weeks time.

Essentially what happened was that we were at the default logging settings and with 40k endpoints, this means we are likely losing event data because the default settings with 40k endpoints is not much.  As a result, we increased logging to what I would consider max entries, but limited by day.  So, 999999999 entries for 2 days.  However, the 2 days thing does not seem to be working.  Am I missing something? 

Looking in the SQL tables and converting the epoch date to real time I have 5 day old timestamps.  I would think that the 2 day limit should kick in and clear these events, but maybe I am not getting it....

I spoke with support as well on this and they said the following:

There are 2 logs for each item because the SEPM writes to 1 until the limit is met according to your settings and then it will switch to the other one.  This data shows that AGENT_BEHAVIOR_LOG_2 has 98,226,581 events and AGENT_BEHAVIOR_LOG_1 has 88,256,401.  The limit is 999,999,999 events or 2 days whichever comes first.  It seems the database is only about 5% of its potential size.  If there was a day of many more events totaling closer to 1 billion then both of the tables could reflect that added size.

 

If your goal is to stop growth then the control log event setting should be less than 100,000,000 and traffic log should be less than 60,000,000 for this environment.  That would help to maintain the database at the current size.

 

If your goal is to capture all the events then the database is definitely going to get much, much bigger.  As I said before, there is a possibility that there are events logged that are unnecessary to your needs.

My thought is that regardless of size, after 2 days events should be cleared.  Much like 10k events for 30 days does not go past 10k events, I would think that if I am at 100,000,000 of 1,000,000,000, on day 3 all events from day one are removed, but obviously not.

You made this change in the SEPM, correct?

It will rollover either after hitting "x" entries or "x"days whichever comes first.

What's the exact version here?

do you have the maintainance plan in place?

SQL 2005/2008 maintenance plan to shrink and limit the transaction logs of sem5 database

http://www.symantec.com/business/support/index?page=content&id=TECH194305

Your interpretation of the settings matches my own.  I was under the impression that any records older than (in your case) 2 days for each specific log type, should get deleted.

To help illustrate the issue, can you actually see these (older than 2 days) logs from the SEPM itself (rather than from SQL)?

Also, as Rafeeq mentioned, you may want to look at how your SQL Server is configured as well.  SQL doesn't actually delete records, it merely frees the record up for overwriting (this is why I'm asking if you can actually see these 5day old results in the SEPM).  The only way to remove a "deleted" record is to shrink the DB (don't do this without prior planning and consulting a DBA).

OK, so I just ran some test reports against the NTP logs.  last 24 hours and then specific date.

If I start today, there are events, but when I get to Friday there was like 11 and then Thursday showed one event, so I would like to assume this means the data is not accessible to SEP and instead means I should figure out why it is not cleaning from the DB to free the space.

As far as version,

Rafeeq, I am forwarding the maintenance plan stuff to my DBA now.  Although, reading it, it does not apply to my version, should that matter?

Just to reiterate, deleting records in SQL does not free up disk space.

This is by design, and only frees up the record to be overwritten.  The only way to free up disk space from deleted records is by shrinking the DB (refer to a DBA first though).

As we've confirmed the records are correctly being "deleted" (i.e. freed up by SQL to be written over with something else), then that means at some point, all the records were active and required (perhaps 5 days ago).  This means there is every chance it will grow to this size again, in which case a shrink will only temporarily free up the disk space.

I'd recommend you contact Symantec (or a partner such as ourselves) to correctly size your SQL server for what you want to retain.

yeah, their sizing tool is crap.  Said maybe 70GB, that's why it makes no sense that it grew like this...

I think my test may be inaccurate to a point.  Last night I ran the DB sweep command because I was fearful of how big things were getting.  We have a SSIM that pulls all the data from SEP DB nightly, so the risk in running the sweep was Sunday's data.

I thought the command failed as the file size was the same, but I just spoke with my DBA and used space is 68GB of 340 GB and growing.  This tells me that the sweep was likely successful.  Before the sweep, the file size was growing like mad, so I know for a fact that the used size equalled the reserve.

At this point I need to wait and just let it grow again to see where it tops out.  Even though the SEP supplied calculator is way off, I suspected double what they suggested, like 150 for the log.  This is why at 300 I grew concerned.  Obviously now that I know the sweep did clear things out I will need to wait again for the dat ato come in and once I am at 2 days, check again to see what it is doing as far as growth.

I am also looking at straight syslog logging to see if that gets me what I need and eliminates my concern of the DB growing out of control.  I see with syslog logging you can filter what is sent out.  Can the same thing be done with the SQL DB logging technique?

That sounds like a solid plan.  I'm afraid at your size of orgranisation, it's always going to be difficult to estimate the DB accurately.

IIRC there is mention of disabling the upload of traffic/packet logs from the clients (as these can be particularly large), but that's dependent upon your usage of course.

As far as your question goe, the filtering option is only available when using the External Logging option.  To control what gets stored in the DB, merely change the thresholds for the offending log type (or disable the upload of the log type from the SEP clients to the SEPM as I mention above).

Just out of curiosity: Do you know the reason of the huge AGENT_BEHAVIOR_LOG_X tables? As far as I know, these tables are used for Application and Device Control, System Lockdown and Tamper Protection. NTP logs are, for example, AGENT_TRAFFIC_LOG_X and AGENT_PACKET_LOG_X (traffic logs and packet log, respectively).

Well, it would seem it is a result of this,

We wanted to log files writtent to USB to see what users are copying off if needed, but it is being used with the defaults.  I am thinking if this is changed to network and removable then it might help things a bit.  Thoughts?

Looking at the behavior logs on the SEPM itself, I see ones that are 3MB, 5MB, 20MB, etc.

The default rule for the "Log files wrtitten to USB" is targetted to only match device ID: USBSTOR*

It looks like this rule of yours has been changed from the default, or is a brand new rule entirely (as the asterisk is also missing).  Is this the case?

Setting this rule to match specific drive types will reduce the amount tracked in comparison to the one screenshotted (assuming you put an asterisk in there) but will actually log more than the default "Log files written to USB drives [AC5]" rule.

How long do these log uploads (assuming that what they are) hang around for?