Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

enormous file CatError.log

Created: 05 Feb 2014 • Updated: 26 Jun 2014 | 20 comments
This issue has been solved. See solution.

In the C:\Program Files\Symantec\Backup Exec\Logs folder of my Backup Exec 2012 server there is a file named CatError.log which seems to be growing without bounds. Right now it is over 2 GB in size which makes it too big to open, and even the tail command crashes on it

What is that file and how can I truncate it to a reasonable size and make sure it doesn't grow to such an extreme size again?

Operating Systems:

Comments 20 CommentsJump to latest comment

Sush...'s picture

Hello Artegic,

    Did anyone enabled the Catalog debug on the server??

 

Please set the following registry keys to the given value and then delete that file (it is the catalog debug log file)

--> set Verbose to 2 in HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Debug.

--> set CreateDebugLog to 0 in HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Logging

 

Thanks,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

Artegic's picture

Hello Sush,

thanks for your quick reply.

It's quite possible that someone enabled various debug settings on this server as it has a long history of troubles.

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Engine\Logging entry CreateDebugLog was already 0.

HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Backup Exec For Windows\Backup Exec\Debug entry Verbose was set to 1, I've changed it to 2 as per your recommendation.

That in itself did not stop the file from growing, and renaming or deleting it failed because the file was still open in Backup Exec Management Service. So I stopped Backup Exec services services, deleted the file and started the services again. Alas, the file CatError.log reappeared and started growing again immediately. Up to now it seems to contain the same line repeating over and over again about twenty times per second with just the time stamp increasing:

[02/05/14 15:28:40]:[12924:10340]:[1]: DynamicQuery::DeleteObject(line 2360) : DynamicQuery::DeletePdi : DSPDIEnumerator::DeletePdi Call Failed EngineError Code -536836945

Any recommendation?

Regards,
Tilman

Artegic's picture

The 2 GB file compressed to 22 MB with ZIP. I transferred this to a Linux system for analysis. Its contents are:

Count Text
13701525 DynamicQuery::DeleteObject(line 2360) : DynamicQuery::DeletePdi : DSPDIEnumerator::DeletePdi Call Failed EngineError Code -536836945
51 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\UpgradeIndexInfo.upg)
47 ODBCConnect::RemoteConnect(line 868) : AllocConnect failed : 0
44 ODBCConnect::DriverConnect(line 1040) : DriverConnect failed with exception : -1 : 0
3 ODBCConnect::DriverConnect(line 1040) : DriverConnect failed with exception : -1 : 183
3 DynamicQuery::DeleteObject(line 2360) : DynamicQuery::DeletePdi : DSPDIEnumerator::DeletePdi Call Failed EngineError Code -536836987
1 THEFHBuilder::WriteCatalog(line 1247) : file(r:\honshu\1798r.amd64\becat\beplugins\thefh\libbasictemplate\memorymappedfile.cpp) function(MemoryMappedFile::alloc) line(1017) error(41)
1 CatRealBuild::StartCatOperation(line 5204) : CATOP_COPY_CATALOG: Unable to retrive catalog information of ImageGuid ( {FED9D0E5-D7E7-401C-8D8E-E140F7937887} ).
1 CatRealBuild::EnableEmulateDeltaCatalog(line 7572) : Cannot open file C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\{D8CB01E0-7238-43D0-BA7D-0818FE6464E3}.prev for read.
1 CatRealBuild::EnableEmulateDeltaCatalog(line 7572) : Cannot open file C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\{B7675655-CE11-4218-8DD1-29CA1B0FD323}.prev for read.
1 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\catstore\{ABEE8EAC-B315-45ED-8E4C-F9610C065657}_1.fh_imgStrings0)
1 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\catstore\{ABEE8EAC-B315-45ED-8E4C-F9610C065657}_1.fh_imgRecord0)
1 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\catstore\{ABEE8EAC-B315-45ED-8E4C-F9610C065657}_1.fh_imgNDMP0)
1 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\catstore\{ABEE8EAC-B315-45ED-8E4C-F9610C065657}_1.fh_imgHeader0)
1 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\catstore\{ABEE8EAC-B315-45ED-8E4C-F9610C065657}_1.fh_imgFile0)
1 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\catstore\{ABEE8EAC-B315-45ED-8E4C-F9610C065657}_1.fh_imgExtraObj0)
1 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\catstore\{ABEE8EAC-B315-45ED-8E4C-F9610C065657}_1.fh_imgDir0)
1 CatFile::Close(line 280) : Fail to delete file(C:\Program Files\Symantec\Backup Exec\Catalogs\GOLGI\{ABEE8EAC-B315-45ED-8E4C-F9610C065657}_1.fh)
1  

(GOLGI being the name of the Backup Exec media server.)

The DynamicQuery::DeleteObject messages started appearing at 2013-10-23T06:45:04, about half a year after the server had been set up. Before that there were only 50 of the CatFile::Close messages and the single THEFHBuilder::WriteCatalog message.

Right now the new CatError.log file has already reached the size of 23 MB again, and AFAICS all of it are those same DynamicQuery::DeleteObject messages that filled the old one.

The server operation does not appear to be affected.

Thanks,
Tilman

Artegic's picture

I had a lengthy WebEx session with a Symantec support engineer on Wednesday who finally told me he had to escalate the case. Unfortunately the newly assigned engineer again wants to talk to me on the phone. I hope we can make an appointment on Monday.

Sush...'s picture

Hello Tilman,

1) Ensure that the Catalog folder is excluded from any Antivirus scanning (On access scan)

2) Also you can stop BE services and delete the Catalog folder. After restarting the BE services it will create a new folder. 

But please note that deleting your Catalog folder will delete all the catalog files and that you will have to re-catalog the tapes / medias to perform the restore job.

 

Else I would suggest you to open a support case with Symantec to troubleshoot this further.

Regards,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

Artegic's picture

Hello Sush,

I excluded the C:\Program Files\Symantec\Backup Exec\Catalogs\ folder from Kaspersky Real-Time Scan, and the messages stopped immediately. Alas they started again five minutes later. I'm now thoroughly confused. Time to open that support call.

Thanks,
Tilman

 

Artegic's picture

I opened a support case via the mysymantec page and got a confirmation E-mail:

Thank you for contacting Symantec Enterprise Support Services.

Case Number 05986116 has been created.

We received your request and are working to grant you access to view and manage your support cases. Once you've received access, you can manage your cases, or create new cases, by visiting http://www.mysymantec.com.

Since then, nothing. Guess I'll have to call them on the phone. (Which I hate.)

Sush...'s picture

Hello Tilman,

   I did check that the technician had tried to contact you last night and again today (2 hours back) but could not reach. Also they have sent emails to you on both the occasions... 

 

Regards,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

Artegic's picture

Hello Sush,

I phoned Symantec support on Friday, had a lengthy discussion with a lady on whether the E-mail address I gave was acceptable, until she finally agreed to attach the case to my account and grant me access to the support portal. So now I can at least see for myself what's going on with the case.

Even though I had explicitly asked to be contacted by E-mail, the technician tried to call me on the phone, using an incorrect number which I don't know where he got it from, then followed up with an E-mail which he sent not to the address I gave either, but to the address of a colleague who was on holiday at the time. "Could not reach" indeed!

I mailed him on Friday giving the correct phone number but telling him I'd be out of the office for the rest of the day and again stating my strong preference to be contacted by E-mail.

Right now I find a "plan of action" saying: "ACTION: Arranged callback on 2/10/2014 as per customer email , should start working on the case". I guess that means I should expect a phone call today. Let's hope I'll happen to be near my phone at the time because otherwise it'll be "could not reach" again.

To be frank, I see some room for improvement in that support process.

Regards,

Tilman

Sush...'s picture

Hello Tilman,

To be frank, I see some room for improvement in that support process.

Point noted. I will try to forward the feedback to the right person.

 

Thanks,

-Sush...

Hope this piece of Information Helps you... and if it does then mark this response as Solution....!!!

sidd3009's picture

Artegic, I'll be looking into it and would arrange a callback for you ASAP. 

Regards,

Siddhant Saini
Advanced Technical Support Engineer, Symantec Corporation 
www.symantec.com

Je_Go's picture

Same error, created support case 06040236

[02/17/14 13:46:33]:[5480:7464]:[1]: DynamicQuery::DeleteObject(line 2360) : DynamicQuery::DeletePdi : DSPDIEnumerator::DeletePdi Call Failed EngineError Code -536836945

Engineer told me this is normal... 

Artegic's picture

Yeah, the first one I spoke to tried to pull that one on me, too. I was able to convince him that a log file growing without bounds at a rate of 25 MB/day constitutes a problem no matter whether the actual log messages appearing in it are or aren't normal. The case has now been escalated.

Je_Go's picture

Changing settings in the Backup Exec Debug Monitor changed the behaviour;

Tools - settings - debug settings:

  • Catalog - unckeck enable catalog debugging outside sgmon

I get entries every 5 minutes instead of every 5 seconds;

 

 

Artegic's picture

The messages I am seeing also occur every 5 minutes, but in a burst of about 400 lines each time, which still sums up to 5000 lines per hour or 25 MB per day. I have disabled everything in Debug Monitor, and support re-checked that first thing every webex session.

The case has been escalated once again last Wednesday and is now, according to the last support person I spoke to, "one level away from Engineering". Haven't heard from the new level yet, today is Friday, so I guess nothing will happen this week anymore.

 

Artegic's picture

It's Friday again. I had a WebEx session with Bkline support this week and uploaded six gigs of logs. (400 megs after compression.) Now I'm waiting for the results of the analysis. Curious what will come out of it.

Artegic's picture

Just a quick update. Last week I installed Service Pack 4 for Backup Exec 2014. The flood of messages in CatError.log continues unaffected.

Artegic's picture

Update: A mere two months after the case was opened, third (or fourth?) level support found out that Backup Exec tried relentlessly again and again to delete a number of folders in a backup to disk storage area that aren't there. The missing folders are named IMG + 6 digits and apparently once belonged to GRT.

Backup Exec's MSSQL database BEDB has a table CatImagefileDeleteTable where it keeps a list of items that should be deleted from the file system. The table is processed periodically, every five minutes as it seems. If a deletion fails, the entry is left in the table to be retried on the next run. Apparently no one saw a reason to treat the case where deletion fails because the item doesn't exist any differently. Result: see above.

The support engineer has now deleted the offending entries directly from CatImagefileDeleteTable using SQL Management Studio, bypassing Backup Exec. This has stopped the growth of CatError.log. He also recommended that I keep a watch on that table to see whether similar entries reappear.

Artegic's picture

Manually clearing the database table CatImagefileDeleteTable has fixed the problem. The CatError.log file has stopped growing, and no new entries have appeared in the table. Support will create a technote.

SOLUTION
Nick Elmer's picture

Symantec strongly recommends against editing the BEDB without the assistance and direction of technical support. There are many reasons why a caterror.log file can grow over time. They include enabled debug log registry settings, unexpected errors, or even corrupt files on disk. For the solution above, the changes were made at the recommendation of support, after consulting the appropriate development group, which determined the appropriate guidance based on the specific errors in the log. If you experience a similar issue with the debug log setting registry values disabled, a technical support case is recommended to identify the underlying cause and recommended resolution.