This issue needs a solution.

Machines stopping because disk full

Created: 27 Mar 2008 • Updated: 12 Jul 2010
RBW's picture
Login to vote
+1 1 Vote
Our servers and workstations are stopping because the disk is full.  The problem appears to be in the folder:
Machine\Program Files\Common Files\Symantec Shared\VirusDefs
Once each minute a new folder is being created with the format tmp????.tmp.  Each folder uses about 15 MB.
We are using version 11.0.1000.1375.
 

Filed Under

Comments

27
Mar
2008

I'm having the same issue and according to other posts this was caused by a bad decomposer but had been fixed back on Feb 20. Has anyone else experienced this and have been able to fix it? Thanks for the help.
27
Mar
2008

Try the doc :

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2007111509244948

Also delete  everything inside "C:\Documents and Settings\All Users\Application Data\Symantec\LiveUpdate\Downloads" folder(Not the downloads folder itself)

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

28
Mar
2008

I have tried these steps but it has not helped, I'm still getting these folders about every minute.
28
Mar
2008

change live update to once every hour in SEPM.
 
This should reduce it alot.
28
Mar
2008

This is set on the Admin tab, correct?
30
Mar
2008

Assuming that the liveupdate is set to daily instead of every four hours(Admin > Servers > Local Site > Edit Site Properties > Liveupdate )

 

Lets do some advanced troubleshooting.

Make sure you take a backup beforehand of anything that we alter (Registry and the database)

Stop the client service by typing "smc -stop" in the run dialog box.

Stop the Symantec Endpoint protection client and the manager service from services.msc

Navigate to "\Program Files\Common Files\Symantec Shared\VirusDefs" and delete everything in this folder including all subfolders.

 

Navigate to "\Documents and Settings\All users\Application Data\Symantec\LiveUpdate\Downloads" and delete everything in this folder including all subfolders.

Database Cleanup

 

Navigate to "\Program Files\Symantec\Symantec Endpoint Protection Manager\Inetpub\content" and delete all the numbered folders (Example: 71219050) beneath each bracketed folder

Example: {1CD85198-26C6-4bac-8C72-5D34B025DE35}

Navigate to "\Program Files\Program Files\Common Files\Symantec Shared\SymcData\sesmvirdef32" and delete all dated folders (Example: 20071231.002)

Navigate to "\Program Files\Program Files\Common Files\Symantec Shared\SymcData\sesmvirdef64" and delete all dated folders (Example: 20071231.002)

Navigate  to C:\Program Files\Common Files\Symantec Shared\SymcData\ and delete the following folders:

sesmipsdef32

sesmipsdef64

sesmvirdef32

sesmvirdef64

Registry Cleanup

 

In the registry, navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\InstalledApps.

Delete these keys

SymcData-sesmipsdef32

SymcData-sesmipsdef64

SymcData-sesmvirdef32

SymcData-sesmvirdef64

In the registry, navigate to and delete the following keys:

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\SharedDefs\SymcData-sesmipsdef32

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\SharedDefs\SymcData-sesmipsdef64

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\SharedDefs\SymcData-sesmvirdef32

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\SharedDefs\SymcData-sesmvirdef64

Start the "Symantec Endpoint Protection" manager and the client service respectively

Start the "Symantec Management Client" Service by typing "smc -start" in the run dialog box

All the folders that we deleted from the database should populate by themselves.

Run Liveupdate from within the management console.

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

RBW
01
Apr
2008

I applied the long solution to our SEPM server and the short solution to several clients.  That appears to have resolved the problem.  Thanks.
01
Apr
2008

Great......:smileyhappy:

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

RBW
02
Apr
2008

The same problem has recurred on machines where I had applied these fixes.  These fixes appear to be temporary solutions.
02
Apr
2008

Maybe something that we are missing over here.
 
Though the issue hasnt re surfaced on the test machine where I confirmed the above mentioned steps.
 
Someone else might like to chip in .
 

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

02
Apr
2008

We followed the above steps, live update had errors afterwards. Running “lucatalog -update" fixed the initial error that was occurring. Now when live update runs, no errors are being displayed, but the defs are not being placed in the "inetpub\content" folders. Forcing a manual update (using the .jdb file) does not work anymore as well.

We did notice that the mfdef25builder.exe process is no longer running after the defs are being downloaded

 

Any help would be much appreciated.


Thanks!

02
Apr
2008

Are the registry entries that were deleted being re-created after the whole process ?
 
 
 

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

03
Apr
2008

Thanks for the quick reply Sandeep.
 
Looking back at the server this morning. I see that most of the folders (6 of the 11) in the "inetpub\content" folder are now updating, but our clients are not updating.
 
For registry keys; in both the installedApps and SharedDefs locations,  the *virdef* keys have been recreated, but the *ipsdef* have not.
 
Looking at the Show Live Update Downloads screen in the console, i'm showing that "2008-04-02 rev. 041" was downloaded this morning at 7:28am eastern. 
 



Message Edited by tom85 on 04-03-2008 07:15 AM

03
Apr
2008

Try deleting the old virus defs on one client and see if that takes it up from the manager.
 

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

03
Apr
2008

Here are the steps we took on the client:


Ran smc -stop
Stopped Symantec Endpoint Protection Service


Deleted numbered folders from
C:\Program Files\Common Files\Symantec Shared\VirusDefs on a client.

Deleted
C:\Program Files\Common Files\Symantec Shared\VirusDefs\definfo.dat C:\Program Files\Common Files\Symantec Shared\VirusDefs\Usage.dat

Deleted everything form.
C:\Documents and Settings\All Users\Application Data\Symantec\LiveUpdate\Downloads

Started Symantec Endpoint Protection Service
Ran smc -start.

 

All files/folders were recreated in C:\Program Files\Common Files\Symantec Shared\VirusDefs.

In the C:\Documents and Settings\All Users\Application Data\Symantec\LiveUpdate\Downloads only the minitri.flg was recreated.

 

The client still shows March 30 r.17 as being the current definitions.

03
Apr
2008

Toying around with the client a little more, we ran across the "C:\Program Files\Symantec\Symantec Endpoint Protection\ContentCache" folder. As an experiment, we followed the same steps as before, but this time also deleted the numbered folders that are inside of the bracketed folders. After this, the virus definitions and other folders were not being populated back on the client.



Message Edited by tom85 on 04-03-2008 09:57 AM

03
Apr
2008

I am outta ideas....
I have been trying to reproduce this, but unfortunately my clients are getting the defs from the manager....
 
To narrow downt try installing(or reinstalling) a new client and delete all the defs in it and see if it gets the new defs from the manager.....
 
 
 

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

03
Apr
2008

This issue is going to be resolved with MR2.  Until then, to deal with the Temp folder creation problem, we've been running a  script created by my coworker on a Fedora 8 Linux box (since deleting folders from a Windows box was slow).  If you have a spare box to install Fedora on, you can use the following script.  I created a folder inside the root profile folder (since the script will have your Windows domain account and password in it) then used the script (shown below the line) along with a file called serverlist containing the list of PCs and servers names.  I created mine by running a "net view >serverlist" at a DOS prompt on my Windows box.  If the PC is offline, it will take a minute or so to timeout and continue on but at least you don't have to babysit it.  When it's done, repeat until MR2 is released.

Note: this is for Windows XP.  Windows Vista has a different path for the virus defs which is located at: C:\ProgramData\Symantec\Definitions\VirusDefs.  We don't use Vista but if you do, you'll need to replace the lines containing

/mnt/folder-name-containing-the-script/Program\ Files/Common\ Files/Symantec\ Shared/VirusDefs/tmp*.tmp

with the following path

/mnt/folder-name-containing-the-script/ProgramData/Symantec/Definitions/VirusDefs/tmp*.tmp


Good luck


________________________________________

#!/bin/bash

inputfile=serverlist
totalnum=$(cat $inputfile | wc -l)

let "current = 0"
cat $inputfile | while read servername
do
let "current = current + 1"
echo Processing $servername \($current of $totalnum\)
mount -t cifs -o user=domain-name/username,pass=password,uid=503,gid=503 //${servername}/C$ /mnt

du -m /mnt/folder-name-containing-the-script/Program\ Files/Common\ Files/Symantec\ Shared/VirusDefs/tmp*.tmp
rm -r /mnt/folder-name-containing-the-script/Program\ Files/Common\ Files/Symantec\ Shared/VirusDefs/tmp*.tmp
umount /mnt
done


03
Apr
2008

Thanks for the reply Bored. I wish the tmp files were the main problem we're seeing. What we're running into now is that the clients are not receiving updates from the SEPM server.
 
We were starting to notice the tmp files, but they weren't the immediate problem on this server. We tried this change to see if it would clear what looked to be a corrupt 32bit virus definition from our server. Before, when we would run Live Update from the management console it would error out when trying to pull that update from Symantec on the 32bit definition(updating the server by using the .jdb file would work fine). After trying this fix, the updates are being pulled by LU without any problems and at the least being partially processed by the server, but the clients are no longer getting their updates from the server. Kind of a 2 steps forward, 1 step back situation for us.  :smileyindifferent:
03
Apr
2008

We just gave it a try reinstalling SEP on a client machine, but it is not pulling updated definitions from the server. It does seem to be pulling it's policy without any issues from the server though.
 
Are there any logs on the server/client that we should be looking that may give us a good starting point of where to begin? From what we're seeing, it's really looking to be an issue on the server side.
03
Apr
2008

Yeah, you are right, It looks like an issue to be on the manager side.
 
Do you have the registry backup before the deletion ?
 
If you do, take the current backup and replace the existing one.
 
Recycle the IIS services and update the content for the client(s)
 
 
 
 

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

03
Apr
2008

Wow, that's really great!
Almost every desktop and server here is getting out of free disk space again
Do Symantec think we are some kind of idiot? I think so, because once more this "antivirus" called SEP is making everybody that work in our company to think that we (IT staff) are idiots and retards.

Let me guess:
 - corrupted definitions?
 - machines with poor performance?
 - definitions updates not working?
 - a lot of applications stopped working?
 - no VPN?
 - antivirus server crashing to death everytime?
 - computer accounts receiving the wrong policies?
Bingo! It's called Symantec Endpoint Protection 11!

But wait, there's a solution for the problem with the VirusDefs folder getting bigger and bigger and bigger and bigger and bigger and bigger and bigger and bigger and bigger and bigger: just run 4356356784365345 different commands on your machine, and it will work for this week!
That's really great, now I just have to repeat it for 800 desktops and 70 servers. Simple, isn't it?!


I've given Symantec enough chances to demonstrate that they could overcome all these basics problems, but I think they don't deserve it anymore. We are just about to call another antivirus vendors, and for the rest of our lives we are going to recommended anyone that we meet to be far away from any product that has the word Symantec on it.

Well, I'll give the LAST chance to Symantec to show that they really care for their clients... I'll be waiting for a  solution for the next 5 days...

Francamente,
Eduardo

03
Apr
2008

We still have the registry backup that we can merge back in, but the content files were deleted yesterday by mistake when we trashed all of the tmp files.  :smileysad:
03
Apr
2008

Content files should not be a problem at all.
 
Lets replace the registry back and see if that is what is causing it.
 

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

03
Apr
2008

While troubleshooting, we found out that our SEPM server was not running MR1. I don't remember this as being listed as one of the bugs that MR1 fixed, but are there any chances that upgrading to MR1 will solve this problem?
 
We currently have a database backup running, as a precaution incase we end up having to start over with this server.
 
After the database finished backing up, we'll give restoring the registry keys a shot.
03
Apr
2008

Check to see if C:\Windows\system32\LogFiles\W3SVC1 is filling up. It was for me.
 
Go to IIS Manager, under properties of the following virtual directories (in your default website) and uncheck Log Visits:
secars
secreg
reporting
content
clientpackages
 
I deleted all but the current log to reclaim GB's of space...still running fine.
04
Apr
2008

Without any further troubleshooting upgrade the manager to MR1.
 
We would discuss it all for sure in case the issue persists.
 

De facto when AV does something, it starts jumping up and down, waving its arms, and shouting...

"Hey!  I found a virus!  Look at me!  I'm soooo goooood!"

RBW
04
Apr
2008

Our version of SEPM was already at MR1.
The problem restarted and then stopped on its own yesterday.  There are some patterns that might help diagnose the problem.
The Application log for the server running SEPM showed the following event at 4/3/08 5:00:01 PM
New virus definition file loaded Version 100403d.
With some exceptions, the repetitive tmp*.tmp folder created started occurring on our client servers and workstations at 4/3/08 5:10 PM.  Folder creation stopped at 4/3/08 9:05 PM and a numbered definition file was created at 4/3/08 9:28 PM.
The Application log for the server running SEPM showed the following event at 4/3/08 9:04:49 PM
New virus definition file loaded Version 100403ag.
The first exception was that the tmp*.tmp files did not exist on the server running SEPM, although they have in the past.  I suspect the program deleted them after successfully updating the definitions. The definitions successfully updated on this machine at 4/3/08 9:04 PM.
The other exception is our 64 bit servers.  None of them have displayed this problem at any time.
I also changed the IIS logging configuration and deleted old log files in c:\windows\system32\logfiles\W3SVC2 which released about 1 GB of disk space.
Kedar Mohile
Symantec Employee
06
Apr
2008

    This issue has been fixed with SEP MR2


Kedar Mohile
Product Analyst
Knowledge Centered Support (KCS) Level-2
Symantec Corporation







Message Edited by Paul Murgatroyd on 04-06-2008 03:17 PM

13
Jun
2008

Maybe, just maybe it is fixed.
11
Jan
2010

Can I move or delete Symantec Shared folder as it covered 38GB

Hi Sandeep/Everybody,

     is there any way to move or delete Symantec Shared folder which is 38 GB covered C drive space in C:\program files\Common files. If yes, we can delete then whether all services would be working fine on SEPM and SEP client.

  If we can move then, how..

Please reply it would be great help for me.. as my server C: drive is almost full now...

Thanks,
Yatendra