Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

lose access to FSA when recalling EV files

Created: 09 Jun 2014 • Updated: 12 Aug 2014 | 25 comments
This issue has been solved. See solution.

This is becoming very frustrating now we tend to lose access to FSA when recalling EV files .. FSA is on version 10.0.3 while Enterprise vault is on 10.0.4

we have enabled passthrorugh recall and relly stuck here 

The server become responsive onlt after the large file complets which could take an hr

Any ideas on how to sole this ???

Operating Systems:

Comments 25 CommentsJump to latest comment

Rob.Wilcox's picture

Always best to have the latest versions, and matching versions.  So first of all update to the latest Cumulative Hotfix for 10.0.4.

Then to the problem - does this happen when you recall *one* file, or when you recall hundreds? What sort of controlled testing have you done?

What OS is the FSA Server?

yobole's picture

I was of the impression windows 2008 FSA have to be on version 10.0.3 as they tend to blue screen on  latest Cumulative Hotfix for 10.0.4.

 when I kill the place holder service the server becomes responsive and noticed 6 download errors totalling 20GB in size 

Rob.Wilcox's picture

Interesting, I wasn't aware of that.  Bit 'rubbish' to say to downgrade, in my opinion..

However, for the original post you are on 10.0.3, and still have issues. Therefore I'd recommend discussing it with Symantec Support. Have you tried that?

yobole's picture

Spoken to Symnect and they keep asking me to downgrade till i dont get issue which i think is a joke 

plaudone's picture

Yobole,

Which version of the CHF is installed on the EV server?  The latest is CHF3.  

Due to changes that were made in the driver it is recommended that the file server be on the latest 10.0.4 CHF3 or an earlier version with pass-through enabled.  This avoids issues that were made to the driver at Microsoft's recommedation.  Those changes have been rolled back in the 10.0.4 CHF3 version of the driver.  

In the earlier version all the volumes that are targets on the file server need to have pass-through recall enabled to avoid the issue with writing to the same location as the original file.  It may be that the large file is not using pass-through, but a Dtrace of the recall would be needed to confirm.  This can also be an issue if the EV server is very busy causing recalls to wait.  Has the server been optimized?  

- Recommended steps to optimize performance on Enterprise Vault (EV), Compliance Accelerator (CA), Discovery Accelerator (DA), and SQL Servers in an EV environment http://www.symantec.com/docs/TECH56172

- TCP Chimney, TCPIP Offload Engine (TOE) or TCP Segmentation Offload (TSO) will cause a transport-level error to be logged resulting in inaccurate hit counts for Accelerator searches: http://www.symantec.com/docs/TECH55653

yobole's picture

Symantec response which I find acceptable. I hace applied the reg key and also configured passthrough recall but I am NOT comfortable downgrading to 9.04.

As discussed, the issue is occurring on Windows 2008 File Servers and currently the FSA Agent is on 10.0.3. I would recommend to downgrade the FSA Agent to 9.0.4 on Windows 2008 File Servers which are having problems. I have seen the server being stable after downgrading the FSA Agent to 9.0.4 and adding the following reg key on the File Servers

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EvFilter\

DWORD value
Enter the name as IncrementVPBCount
Set the value as 1

If you would not want to downgrade the FSA Agent, then we will have to gather complete memory dumps to analyze what exactly the problem is but at this point of time, there is no know solution. The only workaround is to downgrade the FSA Agent to 9.0.4 and add the reg key.

Manoj Chanchawat's picture

Hi Yobole, There can be few reasons for this behavior. plaudone has already detailed couple of them as what may be the possible reason and also changes we have made in 10.0.4 CHF 3 and what you need to do as first steps. Its strongly recommended to upgrade to 10.0.4 CHF3 version as per what is mentioned above. I had worked with Microsoft in past on such cases and will be interested to know if there are still issues related to hang on 10.0.4 CHF3 version. If it does, we can look over that.

Second, about the behavior, EV Placeholder Service will NOT going to wait\hold until whole file gets downloaded. the download of any file in pass-through mode will be in chunks. Also, having a small cache size will affect the performance as it has a threshold for cleanup. After downloading a file, if cache size reaches the threshold, cleanup is trigged. Cache cleanup will delete least recently used files until enough space is freed for the new file.

Third, EV server performance and busyness will also play role in delay!!! 

Once you upgrade and if you still experience the issue, you need do few things: 

  1. Attach the following with Symantec case
    1. full dump of hang
    2. You tried with Pass-through enable or disable
    3. If you are facing this issue with large files or mix of all types of files. 
    4. Cache size
  2. Share the Symantec support case ID so that we can have all the required info
plaudone's picture

The 10.0.2 or 10.0.3 agent should be sufficient on the file server in that configuration as long as pass-through is configured and working properly.  A Dtrace of the Placeholder service would be able to show if pass-through is being utilized when the issue occurs.  

yobole's picture

I have upgraded the EV server and all FSA to EV 10.0.04 CH3 last night 

Rob.Wilcox's picture

Okay, I guess it's a waiting game then now? How often did it used to show this behaviour?

yobole's picture

normally on monday mornings so we just see how it goes in the next coming days ..  I also noticed another FSA making a large number of recalls which might be causing the bottle neck on the master EV server .. cannot seem to find our what is actually making theses recall and I have exclude registry keu polulated with AV and Backup exe ....

However this particular FSA has 3000 orphanned place holders which not sure how to deal with them 

plaudone's picture

We have had customers perform that upgrade and the issue was resolved.  

You can Dtrace the EVPlacholderService on the file server to determine what is making the recall requests.  Will be like the following entry.  Then you can exlclude that application from recalling.  If it comes back with pid:4 then that is a remote request and EV cannot provide the calling application name.  

(EvPlaceholderService)    <2984>    EV:M    WorkItem::GetExeName: The .exe name for for pid: 2696 is Explorer.EXE

For orphaned PH files you can run fsautility -o in report mode to show the orphaned files.  You can also delete them with the same command in normal mode.  

yobole's picture

After the upgrade we has to reboot a windows 2012 FSA twice today after the server became unresponsive 

Eventlogs shows EV error 7206

A locking error has occurred in FileAllocEntry : Too many posts were made to a semaphore.  (0x12a)  
Internal reference: RERL Release Sema4 %4 

plaudone's picture

Has it been determined if there is an application recalling files?  

Would need more info from file server like Event logs and Dtrace to determine what is occurring.  

yobole's picture

seems we have stopped the lose of shares by applying  Changing SMB2 MaxThreadsPerQueue setting to 64 on all FSA

MaxThreadsPerQueue (HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters\(REG_DWORD)

However we are still getting large recalls even by just higlight jpeg files ..

during a Dtrace on the highlighed Jpegs shows

4891        16:36:58.841           [2872]    (EvPlaceholderService)           <3380>    EV:M       [EvPassthruCacheInit] Queueing a data request| FileName:D:\UK\Pro\11030854 Felpham Bognor\Photos\site visit\B.N.N.R.Felpham August 2010\Picture 024.jpg| keyID : 31

4892        16:36:58.841           [2872]    (EvPlaceholderService)           <3380>    EV:L         {PassThroughRecallLimiter::PassThroughRecallLimiter} (Entry)

4893        16:36:58.841           [2872]    (EvPlaceholderService)           <3380>    EV:L         PassThroughRecallLimiter::PassThroughRecallLimiter Caller SID is S-1-5-21-1111383825-1399753330-1979989523-17310

4894        16:36:58.841           [2872]    (EvPlaceholderService)           <3380>    EV:M       WorkItem::GetExeName: Trying to get the .exe name for pid: 3796

4895        16:36:58.841           [2872]    (EvPlaceholderService)           <3380>    EV:M       WorkItem::GetExeNameUsingPHHelper: entry - PID:3796

4896        16:36:58.841           [2872]    (EvPlaceholderService)          <3380>    EV:M       WorkItem::GetExeNameUsingPHHelper: exit - PID:3796, exe name:mcshield.exe

4897        16:36:58.841           [2872]    (EvPlaceholderService)           <3380>    EV:M       WorkItem::GetExeName: The .exe name for for pid: 3796 is mcshield.exe

4898        16:36:58.841           [2872]    (EvPlaceholderService)           <3380>    EV:L         {PassThroughRecallLimiter::PassThroughRecallLimiter} (Exit) Status: [Success]

I have the mcshield.exe in the exculded exe registry on both the placeholder and passthrough entries as we use passthrough recall

SOLUTION
plaudone's picture

yobole,

Thank you for the update on the registry update!  

Did adding mcshield to the ExcludedExes leu stop the recalls from occurring? 

John Santana's picture

So in this case,

The CHF 3 is still the latest for Enterprise Vault 10.0.4 R1 ?

Kind regards,

John Santana
IT Professional

--------------------------------------------------

Please be nice to me as I'm newbie in this forum.

John Santana's picture

Many thanks Rob for the clarification !

Kind regards,

John Santana
IT Professional

--------------------------------------------------

Please be nice to me as I'm newbie in this forum.

plaudone's picture

Yobole,

Do you have anything new on this issue?

Thanks,

Patrick 

yobole's picture

Thank you all for all your help. It looks the SMB settings  MaxThreadsPerQueue setting to 64 on all FSA has stopped the loss of shares and server being unresponsive

The other recall issue seems to have been contained by excluding the PassThrough recall folder from AV. We are now reviewing all our AV settings

Moving forward we are now looking at centralizing EV by removing EV from all file servers and having just one EV FSA server centrally. Thus have an automated process to move files to be archived from all remote sites to this One FSA server centrally. This takes EV away from all remote sites thus making other things like windows journaling needed for Netbackup deduplication to be faster and server refresh less painful  

We then end up with just one massive FSA instead of 30 which will make managing a lot easier. Looking forward to the migration project for this :) as I guess we have a lot of placeholders to recall or move as required .:)

John Santana's picture

Hi Yobole,

Did you just add the entry MaxThreadsPerQueue

or is there anything else that you need to add/change to enhance the stability of the Windows Server 2008 file server like in the following table ?

Parameter Value Default
NtfsDisable8dot3NameCreation 1 2
TreatHostAsStableStorage 1 0
AdditionalCriticalWorkerThreads 64 0
MaximumTunnelEntries 32 1024
MaxThreadsPerQueue 64 20
RequireSecuritySignature 0 0
MaxMpxCt (only applicable to SMB 1 clients) 32768 50

Kind regards,

John Santana
IT Professional

--------------------------------------------------

Please be nice to me as I'm newbie in this forum.

yobole's picture

We just applied the MaxThreadsPerQueue to resolve this issue but looking at the other settings for SMB optimisation

John Santana's picture

Cool, thanks for getting back to me.

So what version of FSA agent installed on your Win 2008 File server ?

According to this article: http://www.symantec.com/business/support/index?page=content&id=TECH185593 the problem is affecting Win2008R2 and EV FSA 9.0.2

mine is Windows Server 2008 SP2 file server (non R2) with the EV FSA agent v9.0.5 CHF 5  but still the problem exist. 

Kind regards,

John Santana
IT Professional

--------------------------------------------------

Please be nice to me as I'm newbie in this forum.