Video Screencast Help

EV10 Placeholder migration - process randomly stops with Error: Enterprise Vault File Placeholder Service not available while processing item

Created: 22 Apr 2013 • Updated: 26 Apr 2013 | 21 comments
This issue has been solved. See solution.

Hi,

We are migrating some placeholders from one volume to another using FSAUtility -pm option

However the process randomly stops with the following error

Error: Enterprise Vault File Placeholder Service not available while processing item \\?\UNC\server.domain.local\archive\file.jpg

Placeholder service is running fine. No errors on EV Server or File server.

Then if we stop the command and run again it allows us to restart the process.

Any idea?

Many Thanks

Operating Systems:

Comments 21 CommentsJump to latest comment

ia01's picture

Anyone please? FSAUtility works fine in test system with small number of files, but in real work where you have millions of placeholders it just doesn't scale out. So much problem with this piece of Utility.

ia01's picture

It does few thousands items and stops and it is repetable after random number of items ...

BTW .. we have 17 million placeholders to move...

Rob.Wilcox's picture

Have you done any monitoring of the processes via Perf Mon?  To see if anything is being leaked? Have you discussed this with Symantec Support?

I can't believe they wouldn't have seen it if is a general problem.

ia01's picture

We also getting these in the logs when running placeholder migration again ...

 

 

        <Item>
          <Item>\\server.domain.local\f$\archive\OOO0015.xls</Item>
          <ErrorCode>Create destination placeholder - Failure: Error accessing the file: \\?\UNC\server.domain.local\archive\OOO0015.xls, error code: 18</ErrorCode>
        </Item>
 
 
We are on EV 10.0.1
ia01's picture

Seems like a memory leak issue. FSAUtility starts with 10MB memory and goes up to 250 and then throws up that error!.... restart the migration starts working again! 

Rob.Wilcox's picture

Which performance counters did you monitor? 'Memory leak' is a little generic...

 

And 250 Mb is a-okay whatever is being monitored, a 32 bit process can go up to 'nearly' 2Gb.

ia01's picture

Now we getting this

 

 

Description:

  Stopped working

 

Problem signature:

  Problem Event Name:        CLR20r3

  Problem Signature 01:        fsautility.exe

  Problem Signature 02:        10.0.1.0

  Problem Signature 03:        4f63b2b1

  Problem Signature 04:        System.Data

  Problem Signature 05:        2.0.0.0

  Problem Signature 06:        4ca2bac8

  Problem Signature 07:        2a8d

  Problem Signature 08:        167

  Problem Signature 09:        System.OutOfMemoryException

  OS Version:        6.1.7601.2.1.0.272.7

  Locale ID:        2057

 

Read our privacy statement online:

  http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409

 

If the online privacy statement is not available, please read our privacy statement offline:

  C:\Windows\system32\en-US\erofflps.txt

 

 

Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled

Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:

Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled

Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:

Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled

Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:

Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled

Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:

Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled

Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:

Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception:Unhandled Exception: OutOfMemo

ryException.

Unhandled Exception:Unhandled Exception: OutOfMemoryException.

 

D:\Program Files (x86)\Enterprise Vault>

ia01's picture

Now getting this while trying to run FSAUtility.exe

 

 

D:\Program Files (x86)\Enterprise Vault>

 

[Window Title]

D:\Program Files (x86)\Enterprise Vault\FSAUtility.exe

 

[Content]

D:\Program Files (x86)\Enterprise Vault\FSAUtility.exe is not a valid Win32 application.

 

ia01's picture

Symantec Tech Support said there is definitely an issue with FSA Utility for EV 10.0.1 and unfortunately there is not Hot Fix available for this. They have Hot Fix for EV 9.0.2 

Now need to plan the upgrade of 12 x FSA servers and 160 File servers to EV 10.0.2 from EV 10.0.1

SOLUTION
ia01's picture

I guess Symantec really need to test all components and functions of EV against a large scale environment before releasing a major version :-)

 

ia01's picture

http://www.symantec.com/business/support/index?page=content&pmv=print&impressions=&viewlocale=&id=TECH190333

 

 

The Enterprise Vault (EV) File System Archiving (FSA) FSAUtility process may return a 'System.OutOfMemoryException' when using the -pm option.

Article ID: TECH190333  |  Created: 2012-06-04  |  Updated: 2012-10-29
Technical Solution for Enterprise Vault for File System Archiving 9.0 9.0.4, Enterprise Vault for File System Archiving 9.0 9.0.3, Enterprise Vault for File System Archiving 9.0 9.0.1, Enterprise Vault for File System Archiving 9.0 9.0.2, Enterprise Vault for File System Archiving 10.0, Enterprise Vault for File System Archiving 10.0 10.0.1

 

Problem

The FSAUtility -pm option may return this condition when processing a large folder structure of items to migrate. 

Error

Error: Exception of type 'System.OutOfMemoryException' was thrown.

 

Event Viewer Log entry

 

Event Type: Error
Event Source: Enterprise Vault
Event Category: FSAUtility Tool
Event ID: 41296

 

Description:
FSAUtility placeholder migration operation from source path \\Source.File.Server\g$\folder to destination path \\Target.File.Server\v$\folder completed with errors. The archive point migration failed to complete successfully, or the database update failed.

 

 

 

If archiving occurs on the destination path, the archived data for that path may become split across multiple archives. Try repeating the migration. Do not archive from the destination path until the placeholder migration operation completes successfully.
 

 

Cause

This is due to the FSAUtility process memory usage increasing until it reaches a critical level on the EV server.

Solution

This issue has been addressed in the following release:

Cumulative Hotfix for Symantec Enterprise Vault (EV) for File System Archiving (FSA) 9.0.4 - FSAUtility processes.

Enterprise Vault 9.0.4 Cumulative Hotfix 1 Release
http://www.symantec.com/docs/TECH199014 

Enterprise Vault 10.0.2
http://www.symantec.com/docs/TECH147789

 

 

Workaround 

Restart the migration using the same command to continue processing the folder structure from the source server. 

Note:  The process will start at the beginning of the folder structure and verify that the items have been moved to the target server.  This will cause a delay in the migration of non-migrated data until the process reaches the point where the data does not exist on the target server. 

 

ia01's picture

Unfortunately workaround is painful. We found that it stops after every few hours randomly and we need to restart the system to get it working again. We have to move 17 million placeholders from one location to another. If we have to sit down and watch the process to fail every day for few days it wouldn't be acceptable.