Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

V-79-57344-33967 on ProgramData\Symantec\CRF

Created: 25 Apr 2014 • Updated: 08 Jul 2014 | 17 comments
This issue has been solved. See solution.

The last run of the self-backup job for my BackupExec 2012 media server produced a bunch of error messages like this:

Datei ProgramData\Symantec\CRF\4f5ea992a6d24e92b9873a39bda85b5f\bin\App_Web_l2vxmiah.dll wurde nicht gefunden, oder es war kein Zugriff möglich.
V-79-57344-33967 - Das Verzeichnis bzw. die Datei wurde nicht gefunden, oder der Zugriff ist nicht möglich.
Element \\GOLGI.artegic.local\C:\ProgramData\Symantec\CRF\4f5ea992a6d24e92b9873a39bda85b5f\bin\App_Web_l2vxmiah.dll wird gerade benutzt - übersprungen.

English translation:

File ProgramData\Symantec\CRF\4f5ea992a6d24e92b9873a39bda85b5f\bin\App_Web_l2vxmiah.dll wasn't found, or no access was possible.
V-79-57344-33967 - The folder or file wasn't found, or the access wasn't possible.
Element \\GOLGI.artegic.local\C:\ProgramData\Symantec\CRF\4f5ea992a6d24e92b9873a39bda85b5f\bin\App_Web_l2vxmiah.dll is currently in use - skipped.

All reported files are within the folder "ProgramData\Symantec\CRF", so apparently they belong to BackupExec itself.

What's happening there? Shouldn't Backup Exec be able to access its own files? Should I worry?

Operating Systems:

Comments 17 CommentsJump to latest comment

lmosla's picture

Hello,

This can occur if there is a an anti virus scan running during the backup. Try running the backup during a different time then the virus scan.

Also make sure there is not another backup program running.

Enable AOFO in the job properties.

see http://www.symantec.com/docs/TECH71644

ArulPrasad's picture

Its not About file access by BE, this might be that file is not present in the location, but the writer has the meta data still not being updated.

Try unchecking the C: and Rechecking the whole C drive in the selection list and see if that helps!

Is this part of warning comes in C: or System state resource?

Regards

Arul

Arul

Artegic's picture

Thanks for the replies. The error has not reoccurred on the following backup run so I guess there's no need to panic. I'm using Kaspersky Anti Virus with hourly updates and scan of critical files after an update (the recommended setting) so it is entirely possible that a scan was under way at the same time as the backup job which reported the error. I searched the Symantec knowledge base for recommendations on which folders to exclude from virus scans but came up blank. Right now I am excluding, in addition to Kaspersky's default exclusions, the Backup Exec deduplication folder because I reckon it might be particularly sensitive to interference from a virus scanner. Would you recommend that I exclude the C:\ProgramData\Symantec\CRF folder, too? What is the purpose of that folder, anyway?

I also inspected the C:\ProgramData\Symantec\CRF folder and found that most of the files Backup Exec complained about are indeed not present now. However Backup Exec explicitly reports them as "currently in use" which seems to suggest they did exist at the time. This is plausible as many of them have "\ASP Temporary Files\" in their paths. Also, the Selection Details view of the backup job's selection list shows just the entire C: drive as selected, without any explicit inclusions or exclusions, which in my book would imply that it should back up everything that exists on C: and not care for anything that doesn't.

AOFO is set to "Use Snapshots", "Automatic Snapshot Provider", and "Process logical volumes sequentially". Backup Exec reports using Microsoft Software Shadow Copy provider without problems.

The errors appeared during the backup of the C: drive. System State backup reported problems.

Thanks,
Tilman

ArulPrasad's picture

well excluding it would turn off SDR capability, not a recommended solution though!!

 would you post a sanitised job log if possible?

Arul

Artegic's picture

Yesterday it happened again. I'm attaching the (German) job log.

Note that the self backup of the media server runs during the day at 1 pm when it is otherwise inactive.

AttachmentSize
joblog.zip 24.75 KB
ArulPrasad's picture

Would yu check if checkpoint restart is Off unde Advanced File option?

Arul

Artegic's picture

I'm attaching a screenshot of the job's AOFO settings.

As the screen is in German, here are literal translations of the settings:

  • "Use snapshot procedure" is ON
  • "Snapshot Provider" is "Automatic - allow selection of snapshot provider by VSS"
  • "Process logical media for backup one after another" is ON
  • "Enable fixed point restart" is OFF

I guess the last one is what you're referring to as "checkpoint restart". So yes, it is Off.

AOFO.PNG
Artegic's picture

The problem currently occurs consistently on all incremental backup jobs (running at 1300h on weekdays) but never on full backups (running typically between 0400h and 0600h on Sundays, depending on the time the preceding jobs take.)

Is there nobody knowing what the C:\ProgramData\Symantec\CRF folder is used for? Backup Exec is the only Symantec product installed on that server, so I don't see how anything else but Backup Exec would create or use a folder below C:\ProgramData\Symantec.

Colin Weaver's picture

I think you probably need to log a formal support case so that we can look into it more thoroughly.

Artegic's picture

Done. Case #06597197. Hope your colleagues find something.

Artegic's picture

Lat week support collected VxGather info and SGMon logs from a failing backup run and announced that the case would probably be escalated. Since then, silence.

lmosla's picture

Looking into this and you should be contacted soon.  

thanks!

lmosla's picture

Artegic,

I reached out to a Duty Manager who is making sure you get a call tomorrow morning EMEA time.

Artegic's picture

Update: The case has been escalated. No solution in sight so far. The current theory seems to be that some registry remnants of an uninstalled nVidia driver are causing this. Support had me manually search the registry for the file and directory names appearing in the error messages. To my untrained eye, the results looked inconclusive, but Back Line Support has been pondering them for a week already without asking me to try something else, so perhaps there's still hope.

Artegic's picture

Update: Today, a WebEx session with BackLine support brought new findings:

The files named "App_Web_*.DLL" which Backup Exec complains about in various directories below C:\ProgramData\Symantec\CRF keep getting created and removed, sometimes several times per minute, with randomly varying eight character alphanumeric strings in the place of the asterisk.

What's more, one place where they are created and deleted again is in individual subdirectories below "C:\ProgramData\Symantec\CRF\ASP Temporary Files\crf\cd095171\882e2271\assembly\dl3" with names consisting of eight random hexadecimal digits (not related in any obvious way to the name of the DLL file.) This is leaving behind a growing series of empty subdirectories. Right now there are already over 32,000 of them.

Obviously Backup Exec is unhappy about seeing files it should backup but having them disappear before it can do so. Also obviously I am unhappy about yet again finding Backup Exec quietly accumulating some dirt in a corner.

Support is now investigating what might be responsible for that strange continuous creation and deletion.

Artegic's picture

Hopefully final update: The support engineer asked me to add the folders

C:\Program Files\Symantec
C:\ProgramData\Symantec

to the exclusion list of Kaspersky Anti-Virus' real time protection. Since then, the continuous recreation of files and directories has stopped and the error message during incremental self backup does not occur anymore.

So lmosla's first hunch was right. The problem was with the virus scanner.

Seems the case is finally solved. Sure, it does make me somewhat uneasy to exclude parts of the filesystem from virus protection. But this machine runs nothing else but Backup Exec and has no users except administrators and backup operators, so I guess the risk is acceptable.

SOLUTION
brentil's picture

We had the exact same issue.  The BackupExec RPC service was using 100% of a CPU all the time no matter what.  It wasn't until I loaded up ProcessMonitor and watched the service that I saw it accessing the C:\ProgramData\Symantec\CRF folder.  It hit that folder about 500,000 times in a matter of minutes endlessly.  I went searching and found this thread so I added the folder to the ignore list in McAfee VSE 8.8 and then the CPU consumption dropped.