Video Screencast Help

Backup exec logs filling up

Created: 21 Oct 2007 • Updated: 21 May 2010 | 19 comments

I am having this problem with backup log files where a certain backup will run and somehow get stuck in an infinite loop which fills up the log file until the hard drive is full. Has anyone had a similar problem? I have all options turned on with the logging and this doesn't happen every time, maybe once a week. I had all optins turned on because I was troubleshooting a problem and wanted to see if a specific file was the culprit, but it wasn't. There's no way for me to figure out what is happening becuase the log file becomes 26 gigs or so and I'm unable to open it, so I have to manually shut down the backup exec services, then delete the file.

Which leads me to my next question/problem. I've tried moving the log files off of the C drive using beutility but it doesn't set that as the new log file directory. It does move the log files, but it doesn't start using that directory as the default directory for log files. Does anyone know if this is a bug or am I not doing it right?



Message Edited by pikes on 10-21-2007 11:38 AM

Comments 19 CommentsJump to latest comment

David Bond's picture
Hi,
 
I have the same problem, I have had it for months now, and I have posted about it a couple of times with no reply to it.
 
Not found an actual solution yet.
 
David
pikes's picture

Can elaborate on what kind of machine you're backing up and what kind of data? Mine is our domain server which is also our exchange server. I'm only backing up roaming profile data (profiles, appdata, desktop, mydocs) and a "homes" folder that contains each users' network drive (for data they want backed up). Like I said earlier, it doesn't always happen, but here is the error I got the last time it got stuck in the loop:

Backup- \\Server\D: New VolumeAOFO: Not enough disk space on the device running the Advanced Open File Option (AOFO) to create or increment the cache file.
To increase the initial size and maximum size of the cache file, use the AOFO Wizard on the Tools | Wizards menu.
Optimum settings for your computer will vary.
V-79-57344-37907 - Snapshot error during backup. Set has been canceled.

Verify- \\Server\D: New VolumeUnexpected end of backup set encountered on Server-Daily.
V-79-57344-34004 - End of data set was unexpectedly encountered.

I know the cache file is big enough. I have the settings set on initial size 5% and max size 60%. The drive has about a terabyte free so I don't think the cache file is reaching 600 gigs unless the infinite loop is causing it. Maybe we can troubleshoot this together and figure out what our problem is.


David Bond's picture
Hi,
 
These are some of my posts, has some info on what i have found out so far
 
 
This happens on 2 different backup servers (we have 3) over 4 different backup jobs(total number of jobs is around 100, inc duplications), i think it is. It doesnt happen all the time, only so far appears to be random. It started a few months back and then stopped and then started again about a month ago.
 
I will post more details when I get time.
 
David
pikes's picture

Ah I see. I was able to look at the log file using command line as you did and here is the first line of the error message up to the repeating "connection lost" message:

</display_contvcb><OperationErrors Type="error">Unexpected end of backup set encountered on Server-Daily.
</OperationErrors><OperationErrors Type="error"><UMI>V-79-57344-34004</UMI> - End of data set was unexpectedly encountered.
</OperationErrors><OperationErrors Type="comm_failure"><UMI>V-79-57344-65072</UMI> - The connection to target system has been lost. Backup set canceled.
</OperationErrors><OperationErrors Type="error">An inconsistency was encountered on the storage media in Server-Daily.
</OperationErrors><OperationErrors Type="error"><UMI>V-79-57344-33994</UMI> - The data being read from the media is inconsistent.
</OperationErrors><OperationErrors Type="comm_failure"><UMI>V-79-57344-65072</UMI> - The connection to target system has been lost. Backup set canceled.
</OperationErrors><OperationErrors Type="comm_failure"><UMI>V-79-57344-65072</UMI> - The connection to target system has been lost. Backup set canceled.

etc, etc..

It has happened 2 times in a row now. However, I was able to run another job that does an incremental on Sunday that completed with exceptions; something about quiet time, an error I've had before and I'll solve again later. The daily backup failed the same way on Friday night and Monday night.

 Anyway, one thing I can remember about the server in question is that I had to manually install the remote agent and AOFO on the server because when I tried to do it remotely it said "access is denied" even though I tried doing it as the domain administrator.

pikes's picture

After looking at the server event logs, each time it goes into that loop it has crapped out on a file saying:

"The operating system returned an unusual error while backing up the following file:

 \\server\\file.whatever

 It is possible that this file is incomplete and therefore should not be restored. "

It's a different file each time too.


I searched the net for the error message and came up with this:

http://seer.support.veritas.com/docs/277891.htm

According to that link:

"According to MS articles 312362 and 304101 this error is caused by a lot of files opened (more than the cache manager can handle) and as a consequence the cache manager has exhausted the available paged pool memory. By default the memory manager tries to trim the allocated paged pool memory when it reaches the 80 percent of the total paged pool. Lowering this threshold from 80 percent down to 40 will avoid those situations where the memory manager is struggling to keep up with the paging demand."

This may or may not be our problem because the error message they show is in backup exec while my error message is on the servers' application event log.

Do you run any local backups on the server using ntbackup.exe? I am backing up (exchange, C drive) (locally) using it at the same time backup exec is running but they aren't backing up any of the same files.

It could be a VSP error as well. Do you use any AOFO options?

See this thread also: https://forums.symantec.com/syment/board/message?b...

Obviously we have the same error message, probably for the same reason. If we can figure out what our setups have in common maybe we can find out exactly what is causing the problem!

David Bond's picture
Hi,
 
I use AOFO (VSS for 2k3 and VSP for 2K) The servers that have the problem (from what i remember) are both 2k3 and 2k server. I will have to wait and see which servers exactly were causing the problem, as when it happens it screws up all my backups as it fills the drive so all the backups arent added to the BE database, its on the same partition as the logs.
 
It will probably happen this friday, when my full backups occur.
 
David
pikes's picture

So it only happens when you run full backups? Hmm... are you backing up a lot of small files? I mean, lots of ~1kb files.

David Bond's picture
Yep, on some of the servers, there are around 15m files, around 2-3KB in size. But I will have to see if it happens tomorrow and which servers it happens on to see if it is one of these servers that causes it, but i think it is, as i remember thinking it could be due to the number of files and its listing everyone in the log (but turned out it was just repeating an error message, as you know :) ).
pikes's picture

So last night, the actual backup ran fine but the copy to the off site backup to disk folder had that infinite loop problem again with the same error message.

After each daily backup finishes, we encrypt and copy the backup to an external USB HDD that is taken off site each day. I noticed that the copy going to the off site backup was actually bigger than the original backup when I canceled it. The original backup was 27 gigs while the offsite backup was at 43 gigs when I canceled it.

David Bond's picture
Exactly the same as me, which it why it only happens on fridays, I only duplicate the full weekly backups at the moment, except for the backup server and CPS, which is done every day.
 
My duplications go to tape and are encrypted also (256bit).
 
And i noticed that it also writes more to the tapes than was actually backed up.



Message Edited by David Bond on 10-25-2007 10:28 PM

pikes's picture

Excellent, so, can we get a symantec employee to help us on this one? Since there's obviously a bug with the program (at least, that's what conclusion I have come to from the information we have gathered) I hope we can get some help.

pikes's picture

Bump, any symantec employee response on this?

pikes's picture

So I deleted that encrypted copy backup from the template, and it ran fine for a few days, so I re-added it. It's currently working great. Have you tried deleting the copy part from the template, then re-adding it?

David Bond's picture
I cant remember if I have, but last week (I was was off ill) was fine, I didnt get an error onthe duplication. This has happend before, it started doing it, then stopped, then it started again, and now on one of the servers its stopped again. The other server, it just isnt backing up right now, all my backups just started to fail, so looking in to why that is. Also the duplications from the failed backups, have locked up, saying they have completed, but are running, and cant be cancelled. So just about to restart the server.
 
Edit: Dont have to restart it now, the cancellation worked after a day (i cancelled them yesterday) and now they have just stopped.



Message Edited by David Bond on 11-06-2007 06:50 AM

David Bond's picture
Just happened again, its strange, it happened on both backup servers again at the same time. Both due to duplications, both failed to complete their backups, and both screwed up and their duplicatuions of the incomplete backups by backing up far more data than was was actually backed up. One of them backed up only 11GB, but i cancelled the duplication at 880GB
pikes's picture

Ouch! I haven't had it happen since I re-created that part of the template. I doubt we'll get this resolved unless one of us goes to paid support unfortunately :(

David Bond's picture
I will recreate the duplications, see if it helps. I doubt we would get it solved even if we went the paid support route, based on the descriptions you get here from other peoples experience with it.