Endpoint Protection

A few weeks ago, a virus hit us, and we're still finishing cleanup.  Unfortunately, our reporting server isn't a lot of help because it shows bad or incomplete data on the dashboard.  We're running Symantec Antivirus 10.1.5.5002 and Symantec had us install the reporting server on MSDE for simplicity.  (Before you chant 'SEP11,' I'd been recommending it for many months.)  Our server manages about 1200 nodes.  Full reports appear to show good data, but the Home dashboard has some issues.

Issues identified:

• The "Risks by Server Group: Past 24 Hours" group is blank, reads 'Nothing to display'
• The "New Risks: Past 24 Hours" group has no data, only says 'No new viruses in the last 24 hours'
• The "Risks Per Hour: Past 24 Hours" graph has no data over the past 24 hours, top of the y axis is 0
• Reports > Risk reports > Risk Distribution Over Time, Monthly interval, All products, Specific dates, 12-15-09 to 1-15-10 returns a report labeled 'Monthly Distribution for December 2009', shows a huge outbreak beginning December 28th through the 31st (204 risks, then 215, then 988, then 36 -- and we only had 800 managed nodes at that time; I was not the security administrator three weeks ago)

Does anybody know what might be the cause for these issues on our reporting server?  I've tried to contact support, but with such long hold times and dropped calls, I've had to focus on virus topics rather than reporting.

The virus alerts that I do receive have timestamps that vary by around two weeks.  Is this normal?  See below for examples.

The cause was an extremely large log file.  Here is the process I went through to troubleshoot.  It should be helpful to you regardless of what issue you are seeing with your reporting server.

1. Did reporting server work previously?  If it has never worked, refer to KBs on installing reporting server and troubleshooting it.  (They walk you through confirming the DB, checking .php, checking IIS.)  For me, because reporting server already was working, these weren't helpful. I could see my Reporting server, I just wasn't seeing the right data.
2. Is the Reporting Agent service running?  If not, start it.  If so, you still may wish to Restart it so that you know it's not hung on something.
3. Is the Symantec Antivirus service running?  If not, start it.  You still may want to Restart it.
4. Are files located in C:\Program Files\Symantec\Reporting Server\Upload?  If files are not coming in here, clients may not be reporting events to the server.  Check your reporting settings in the System Center.
5. Are files coming into C:\Program Files\Common Files\Symantec Shared\Reporting Agents\Win32\Temp.LogReader and then being cleared?  If it's empty, it probably means it's working properly.  If it's empty, that probably means it's working.
6. Is there a ridiculously large log file in C:\Documents and Settings\All Users\Application Data\Symantec\Symantec AntiVirus Corporate Edition\7.5\Logs, or is a log file out of place?  In my case, a log file from the day I set up Reporting server was 83MB and named 01072010.  The next log files in the sequence were 01202010 and 01212010 and 01222010.  Symantec couldn't finish processing the log because it was so huge, or corrupt, or both.  I deleted the 010702010 log, then modified a file to point to the correct log -- more on this in a moment.
7. Does C:\Program Files\Common Files\Symantec Shared\Reporting Agents\Win32\logsender5.stat point to the correct log file?  If your logs have not been processed and you've just deleted a corrupt log, the first line should read LOGFILE = mmddyyyy where mmddyyyy correlates to your oldest log file.  This is because you need to start processing your oldest logs first.  Symantec will move through to the newest.  The second line should read LASTLINE = 1.  Or 0.  Basically it instructs Symantec to start reading your old log file at line 0 or 1.  If reporting is working correctly, the first line should read LOGFILE = mmddyyyy and match today's date, the most recent log file.  The second line reads LASTLINE = x, where x a line near the end of the current log file.  This is where Symantec left off the last time it processed the log.  This won't necessarily be the last line in the log file at this moment, but it should be close.  If you look here and find you need to modify the file, I recommend making a backup.  You will need to stop the Reporting Agent service in order to modify this file.
8. Check log files for clues of what might be wrong.  The log files are: LogReaderEvents_yyyy-mm-dd.log, Parent_Inv_yyyy-mm-dd.log, ReporterSvc.log, LogReaderInventory_yyyy-mm-dd.log, viruscat_yyyy-mm-dd.log, logsender_yyyy-mm-dd.log, notag_yyyy-mm-dd.log, history_yyyy-mm-dd.log, Backup_yyyy-mm-dd.log, DBmaint_yyyy-mm-dd.log.  All of these are located in the same directory as logsender5.stat.  Errors are explicitly stated as ERRORs.

Again, in my case, I stopped the Reporting Agent service, deleted the oldest log file which was very large, then modified my logsender5.stat file to point to the oldest remaining log file, starting on line 1, then restarted the Reporting Agent service.  Logs began processing normally within 10-15 minutes.  (Then overnight I received 1500 notifications.)

Hope this helps someone.