Video Screencast Help
Symantec Appoints Michael A. Brown CEO. Learn more.

High disk activity by SEP services

Created: 15 Mar 2013 | 9 comments

Have SEP 12.1.2 with latest updates on SBS 2003 server w/15 users. Finally got rid of problem with runaway sem5.log size by completely uninstalling SEP, doing fresh install w/new database, and then forcing client reconnects. Restoring original database from backup leaves log problem. However I still have problems with runaway disk utilization by SEP processes. Problem became noticable several weeks ago, and has grown progressively worse. PerfMon Avg Disk Queue Length periodically pegs out at max for long periods, eventually forcing other client apps using the server to time out. Config Task Manager for I/O Read & Write bytes shows massive utilization by LUCallbackProxy.exe, SemSvc.exe and dbsrv12.exe. For example, within 10 minutes, LUCallbackProxy will read 1,409Mb and writes 939Mb. SemSvc reads 1,023Mb and writes 474Mb. These are WAY beyond numbers of "real" applications on same server by orders of magnitude - even heavily used shared file databases. Eventually, dbsrv12 collects even more over time, reading 566Mb and writing 1,190Mb by 30 mins (and keeps going). Stopping all SEP services allows Avg Disk Queue Length to eventually return to normal, but of course no protection then. There are also multiple instances of LuCallbackProxy (at least 3), although only one will have high disk I/O. These are insane amounts of disk activity and server resource utilization. I have read several dozen forum technotes trying to find answers, but no solutions so far. How can we get this resource utilization under control?

Operating Systems:

Comments 9 CommentsJump to latest comment

pete_4u2002's picture

whats the memory on the server?

may be you want to increase

TekNikLee's picture

The memory is 4Gb, but seriously, the problem is not cache size but data volume.  This problem became obvious in SEP during the inability to truncate the sem5.log file, which writes out at a rate of a gigabyte per day - with only 13 licenses.  Within a couple of weeks, the log would fill all of the System drive (about 20Gb free in our case), and had to be manually deleted.  I can't even imagine that much log activity on a server log with a few hundred licenses.  The fact is that EACH of the three Symantec processes are generating more data volume than all other server processes COMBINED.  Are we the only customers experiencing large volumes of periodic Symantec related disk activity?

LinwoodR's picture

We are seeing the same issue on our 2 SEP Manager servers.  Heavy C: drive disk activity reading  definitions from inetpub\content folder, presumably to send to clients.  We have just spread out content distribution to be a 2 hour window, so should not see a crush at any one time, but this did not solve the problem.  Access rates are excessive and are affecting other production servers with major disk latency issues.  All our servers are under VMWare with disks in SAN. 

TekNikLee's picture

I'm curious if your activity also grew progressively worse over several months, and if the same processes are responsible for the disk activity (add I/O Read & Write bytes columns to Processes tab under Task Manager).

LinwoodR's picture

Yes it did grow worse over the last few weeks. Excessive disk activity on SEPM was always read activity to the C: drive content folder where the definition files are located. SEPM Network traffic at the same time was outbound, heavy, and to a variety of clients.  This is consistent with the times that definition updates were being sent to the clients.    Access by SEPM to its C: drive was directly correlated to the increased disk latency experienced by all servers in the VMWare cluster. 

Things we've seen and changes we've made:

1.We recently added new production servers to the VMWare cluster which may have overcommitted  resources. We are moving guests to other clusters and shutting down some unused guest test systems.

2. The SEPM C drives were on the low-speed disks in the SAN.  They have been moved to the high-speed disks. 

3. Recent SAN firmware upgrade might have caused performance issues with VMWare. Have applied VMWare fixes and updated VMTools on some guests.  That has helped.  Will roll to others.

4. A few SEP clients were continually asking for downloads.  Reinstalled the clients and problem disappeared.  We know that some client information about definition release levels is not being properly updated in the SEP DB due to bugs in 12.1.1 and 12.1.2.  I can't help but wonder if that is why definitions keep being delivered to PCs that don't actually need them.  That is something that would get worse over time.  Watching Resource Monitor on SEPM, I was surprised to see how many 'full' definition files were being delivered rather than delta definition files. (We keep 15 prior releases of definitions on SEPM).

Most of our latency issues have settled down now. 

TekNikLee's picture

Out of curiosity, why are ANY previous versions of defination files needed? Is there any advantage to having more than one version, and is this something that can be modified?

pkecun's picture

Experiencing the same here..SEP IO Write Bytes.jpg

..that's an SBS2011 box with SEP 12.1.2 - I had to pause for a moment when I saw semsvc.exe had written more than every other process combined over a 3 day period. 40ish clients and checking for definitions every 4 hours. That seems a little overkill to me. If it's considered normal I think I'll be looking at moving SEPM to a dedicated box.

 

Mithun Sanghavi's picture

Hello,

I would suggest you to Create a Case with Symantec Technical Support.

How to create a new case in MySupport

http://www.symantec.com/business/support/index?page=content&id=TECH58873

Phone numbers to contact Tech Support:-

Regional Support Telephone Numbers:

  • United States: 800-342-0652 (407-357-7600 from outside the United States)
  • Australia: 1300 365510 (+61 2 8220 7111 from outside Australia)
  • United Kingdom: +44 (0) 870 606 6000
 
Hope that helps!!

 

Mithun Sanghavi
Senior Consultant
MIM | MCSA | MCTS | STS | SSE | SSE+ | ITIL v3

Don't forget to mark your thread as 'SOLVED' with the answer that best helped you.

theexplorer's picture
 
High CPU utilization or disk I/O during virus definition update on drive with multiple virtual machines.
 
http://www.symantec.com/docs/TECH106373

Good Luck!