I am wondering if anyone else has experienced problems with drives going down and showing as AVR mode after changes to zoning on the SAN.  We have EMC CX500 SAN and the Netbackup environment consists of 1 master server, 2 media servers and 8 san media servers.  These are sited evenly across two data centre sites served by two robotic tape libraries on either site with 8 LTO3 drives in total.  We are using SSO and all kit is connected by fibre with Qlogic HBA's. 

 

Recently after actioning changes to zoning on new servers to have visibility of the tape libraries and cross site zones,  we have noticed problem with Netbackup,  where drives have gone down/missing and showing as AVR when NBU services have been stooped and started.  The problem does not happen immediately after zoning changes have been committed and jobs do complete overnight,  but the problem becomes evident when running new backups.  Jobs show as running and then mounting but just hang indefinitely at this point.  We have undone the changes made to zoning and this has not necessarily resolved issues, the kit in the SSO environment has been rebooted and still no better.

 

Drives continously show as AVR and the device manager service on the master struggles and fails to start and shows device active for disk only.  The disk based jobs are unaffected and after a lengthy wait the device manager service does start on the master. When running the scan command we get drives missing or changers even though the operating system can see everything correctly in device manager.  All systems run under Windows Server 2003 SP1 with NBU 6.0 MP4.  Symantec have assisted but nothing conclusive has been identified as root cause, after 24-48 hrs system just seems to come backup as normal after running the device configuration wizard.  Even drives that is reports missing in the device confiuration wizard show up eventually.  I am just wondering if certain servers which we haven't been able to reboot with them being high availabilty production systems may have some odd condition on them causing the rest of NBU to hang up.  Its as if there is something that times out after which everything starts working.  No other devices that run off the SAN or fibre are affected,  just NBU and I am wondering if a HBA SCSI reservation can cause this or is it to do with a reset that may ocurr on the fibre switches/routers when changes are commited.

 

In the recent incident we also came across four tapes stuck in drives even when no job failures had been reported and nothing was showing as active.  Its as if they had been caught on the on the end of a backup and at the time the changes had been made to the zoning.  Resetting the drives and rebooting the libaries did not release the tapes and in the end physically pressing the eject button on the drive sleds resolved the problem.

 

Everything is working fine at present but I would like to know if anyone has come across similar issues and how they have managed to resolve.