Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

BE2012 only uses 2 tape drives in Virtual Tape Library.

Created: 18 Jul 2012 • Updated: 19 Jul 2012 | 11 comments
This issue has been solved. See solution.

Background:

Our BE2012 connected with physical and vitual tape library (8 drives), we backup the data to VTL daily and export the monthly backup set to physical tape.

We've updated the BE2012 with Service 1a two weeks ago, everything goes fine. 

OS: Windows 2008 R2

Physical Library: Hewlett Packard MSL G3 Series Library. 2 slot

Virtual Tape Library: DataDomain DD670, 8 drives

No specific drives selected in all backup jobs, we checked all the drives in VTL.

Issue:

Last week, we encountered an issue about BE2012. the BE2012 only uses 2 drives only in VTL. Hence some of backup jobs keep queued and wait until the one of first two backup jobs completed. And the job status report "Ready no idle devices are available...."

We've called the support last Friday, and they finally replied me yesterday, what they did is try to remove the Windows Driver "Hewlett Packard MSL G3 Series Library" and use "Unknown Medium Changer" under "Device Manager" AND send me an email about close this case! 

After checking backup job run on yesterday, the same issue happened.

This morning, I've tried to remove that VTL, restart server again. BE2012 recognized the VTL and we configurated it again.

When we try to inventory the tapes in VTL, the BE2012 inventory process still use 2 drives in VTL.

Anyone have idea about this?

Now we have to move two backup jobs which takes over 12 hour backup to BE2010 server, however, this is only a short term solution for us.

BE2012 gives me a lot of time on troubleshooting and calling Symantec support. I miss BE2010!

Comments 11 CommentsJump to latest comment

Kiran Bandi's picture

1) If unlimited drive option is not working then BE should only use one drive within VTL.

2) You have LEO option, for your HP tape library.

I doubt BE is messingup both these options and seems like it is applying LEO to VTL, because all the drives are reported as online but only two are being used.

If i were you, i will try removing and readding VTL unlimited drive license. Remove license and see how many drives BE will use before adding it back. If it uses two drives, then remove LEO license and add VTL unlimited drive option and check. 

Benjamin Chiu's picture

I tried to remove, unfortunately, BE2012 prompted "Product licenses may not be deleted"

Kiran Bandi's picture

Follow the procedure mentioned in this TN to do so: http://www.symantec.com/docs/TECH181692

Benjamin Chiu's picture

I tried to unselect the license and reboot the server.

The result still the same, the BE2012 only use the 6th and 7th drive in the VTL.

Benjamin Chiu's picture

I tried to create a new backup job and config this backup job use specific drive.

However, it result still as same as before, the job state is "Active" but status is "Queued". The tapes in storage are Scratch Media. After checking the Backup Exec Debug Monitor, I find the following

PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_INFO )
 
            Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}
PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_DEVICE_GUID )
 
            Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}
 
            TargetDevice = {B1F45157-B63E-445A-B635-F29CD2B099EE}, "Robotic library 0001 [0460..0521]" (Device Pool)
 
            TargetDevice = {C09ACD21-FAB3-4DAC-88E6-F4182F85DE29}, "Tape drive 0001" (Drive)
 
            TargetDevice = {325E172C-0B98-49B3-9C0E-8D2B5E5C5C03}, "Tape drive 0002" (Drive)
 
            TargetDevice = {CE04895D-73C5-4D29-B53A-0EB9E51A9E71}, "Tape drive 0003" (Drive)
 
            TargetDevice = {67C0AEFF-CD62-49D6-B9FE-CA2473516DBD}, "Tape drive 0004" (Drive)
PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_MEDIA_GUID )
 
            Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}
 
            TargetMedia = {00000009-03E9-0000-0000-000000000000}, "Keep Data for 4 Weeks" (Media Set)
PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_EXCLUDE_MEDIA_GUIDS )
 
            Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}
PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::CheckTargetDeviceStatus()
 
            Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}
 
            Job = {04D7CFF8-7808-4EC2-A5D0-EC73BBB52883}, "HKDC-BACKUP01 Backup 00218-Full"
 
            TargetDevice = {C09ACD21-FAB3-4DAC-88E6-F4182F85DE29}, "Tape drive 0001"
 
                State = 0003, ReservationID = 0000, AvailableMovers = 0001
 
            TargetDevice = {CE04895D-73C5-4D29-B53A-0EB9E51A9E71}, "Tape drive 0003"
 
                State = 0003, ReservationID = 0000, AvailableMovers = 0001
 
            TargetDevice = {325E172C-0B98-49B3-9C0E-8D2B5E5C5C03}, "Tape drive 0002"
 
                State = 0003, ReservationID = 0000, AvailableMovers = 0001
 
            TargetDevice = {67C0AEFF-CD62-49D6-B9FE-CA2473516DBD}, "Tape drive 0004"
 
                State = 0003, ReservationID = 0000, AvailableMovers = 0001
PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::MountOverwriteMedia() - 0004 drives
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - 0260 media view overwritable media
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - 0046 initial priority overwritable media
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - 0046 priority overwriteable media after blank/unknown
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - 0046 priority overwritable media after standalone
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - qualified drive Tape drive 0003 slot 0467
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - reserve error ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - qualified drive Tape drive 0002 slot 0467
BECAT:    [07/19/12 11:39:12] [6012]     SQL connection: 790fdb0 select * from ControlInfo
BECAT:    [07/19/12 11:39:12] [6012]     SELECT * FROM CatImageFileDeleteTable
BECAT:    [07/19/12 11:39:12] [6012]     SQL connection: 790fdb0 SELECT * FROM CatImageFileDeleteTable
BECAT:    [07/19/12 11:39:12] [6012]     result(0) numRetry(300)
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - reserve error ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - qualified drive Tape drive 0004 slot 0467
BECAT:    [07/19/12 11:39:12] [6012]     Number of records returned: 0
BECAT:    [07/19/12 11:39:12] [6012]     Cleanup images : 0
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - reserve error ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - qualified drive Tape drive 0001 slot 0467
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - reserve error ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)
PVLSVR:   [07/19/12 11:39:12] [7356]     PvlSession::MountOverwriteMedia() - ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE) - 0000 seconds
PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_MOVER_MOUNT_OVERWRITE )
 
            Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}
 
            ERROR = 0xE000810C (E_PVL_DRIVE_NOT_AVAILABLE)
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.720 [server]             + storagedevice.cpp (345):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.720 [server]             | DoMount Execute returned: 0xe000810c
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.720 [server]             + backupjob.cpp (6013):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.720 [server]             | BackupJob: unable to scratch mount, will retry append later.
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.720 [server]             + backupjob.cpp (6126):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.720 [server]             | BackupJob: unable to mount, error 0xE000810C.
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.721 [server]             + storagedevice.cpp (516):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.721 [server]             | Closing out mover session(0000000009CAF760): {253da68d-c69c-43ca-b71e-5df5c41703d9}
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.721 [server]             | device id: {c09acd21-fab3-4dac-88e6-f4182f85de29}
PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_MOVER_DISMOUNT )
 
            Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}
 
            Flags = 0x00000020
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.721 [server]             + storagedevice.cpp (540):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.721 [server]             | Dismounted mover(0000000009CAF760).  result - 0x00000000
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.721 [server]             + storagedevice.cpp (553):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:12.721 [server]             | Releasing mover: 0000000009CAF760
PVLSVR:   [07/19/12 11:39:12] [7356]     AdammSession::Release()
 
            Session = {253DA68D-C69C-43CA-B71E-5DF5C41703D9}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:12.722 [server]             - DeviceManager: incoming event fired
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:12.722 [server]             - DeviceManager: Executing RequestDeviceCall from thread 1520.
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:12.722 [server]             - DeviceManager: stalling
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:12.722 [server]             - DeviceManager: going to sleep for 29000 msecs
BESERVER: [07/19/12 11:39:16] [2232]     -1 SecurityBO: Successfully setup RAWS certs.
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.724 [server]             - DeviceManager: timeout event fired
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.724 [server]             - DeviceManager: processing pending requests
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.724 [server]             - DeviceManager: Processing device request for thread 1520.
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.724 [server]             - DeviceManager: query cache miss
PVLSVR:   [07/19/12 11:39:41] [7356]     PvlEntityDatabase::QueryEntity()
 
            ParentEntityType = DRIVE_POOL_ENTITY_TYPE
 
            ParentEntityGuid = {B1F45157-B63E-445A-B635-F29CD2B099EE}
 
            EntityType = NULL_ENTITY_TYPE
 
            EntityGuid = {00000000-0000-0000-0000-000000000000}
 
            EntityName = ""
 
            EntityFlags = 0x00000000
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             - Available and held drives from the pool
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0001 {c09acd21-fab3-4dac-88e6-f4182f85de29}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0002 {325e172c-0b98-49b3-9c0e-8d2b5e5c5c03}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0003 {ce04895d-73c5-4d29-b53a-0eb9e51a9e71}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0004 {67c0aeff-cd62-49d6-b9fe-ca2473516dbd}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             - Drives not in use by the Engine
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0001 {c09acd21-fab3-4dac-88e6-f4182f85de29}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0002 {325e172c-0b98-49b3-9c0e-8d2b5e5c5c03}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0003 {ce04895d-73c5-4d29-b53a-0eb9e51a9e71}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0004 {67c0aeff-cd62-49d6-b9fe-ca2473516dbd}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             - Drives available (not in use and not held)
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0001 {c09acd21-fab3-4dac-88e6-f4182f85de29}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0002 {325e172c-0b98-49b3-9c0e-8d2b5e5c5c03}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.730 [server]             -     Tape drive 0003 {ce04895d-73c5-4d29-b53a-0eb9e51a9e71}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.731 [server]             -     Tape drive 0004 {67c0aeff-cd62-49d6-b9fe-ca2473516dbd}
PVLSVR:   [07/19/12 11:39:41] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_DEVICE_GUID )
 
            Session = {18606A09-966C-4049-884D-FCF602B38D5D}
 
            TargetDevice = {B1F45157-B63E-445A-B635-F29CD2B099EE}, "Robotic library 0001 [0460..0521]" (Device Pool)
 
            TargetDevice = {C09ACD21-FAB3-4DAC-88E6-F4182F85DE29}, "Tape drive 0001" (Drive)
 
            TargetDevice = {325E172C-0B98-49B3-9C0E-8D2B5E5C5C03}, "Tape drive 0002" (Drive)
 
            TargetDevice = {CE04895D-73C5-4D29-B53A-0EB9E51A9E71}, "Tape drive 0003" (Drive)
 
            TargetDevice = {67C0AEFF-CD62-49D6-B9FE-CA2473516DBD}, "Tape drive 0004" (Drive)
PVLSVR:   [07/19/12 11:39:41] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_MEDIA_GUID )
 
            Session = {18606A09-966C-4049-884D-FCF602B38D5D}
 
            TargetMedia = {00000009-03E9-0000-0000-000000000000}, "Keep Data for 4 Weeks" (Media Set)
PVLSVR:   [07/19/12 11:39:41] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_EXCLUDE_MEDIA_GUIDS )
 
            Session = {18606A09-966C-4049-884D-FCF602B38D5D}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.739 [server]             - DeviceManager: Handing out session {18606a09-966c-4049-884d-fcf602b38d5d} based on available drive:
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.739 [server]             -     Tape drive 0001 {c09acd21-fab3-4dac-88e6-f4182f85de29}
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.739 [server]             - DeviceManager: returned result to caller 0x00000000
BENGINE:  [07/19/12 11:40:35] [1328]     2012-07-19T11:39:41.739 [server]             - DeviceManager: going to sleep for 900000 msecs
PVLSVR:   [07/19/12 11:39:41] [7356]     PvlEntityDatabase::QueryEntity()
 
            ParentEntityType = DRIVE_POOL_ENTITY_TYPE
 
            ParentEntityGuid = {B1F45157-B63E-445A-B635-F29CD2B099EE}
 
            EntityType = NULL_ENTITY_TYPE
 
            EntityGuid = {00000000-0000-0000-0000-000000000000}
 
            EntityName = ""
 
            EntityFlags = 0x00000000
PVLSVR:   [07/19/12 11:39:41] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_JOB_INFO )
 
            Session = {18606A09-966C-4049-884D-FCF602B38D5D}
 
            Job = {04D7CFF8-7808-4EC2-A5D0-EC73BBB52883}, "HKDC-BACKUP01 Backup 00218-Full"
PVLSVR:   [07/19/12 11:39:41] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_MORE_JOB_INFO )
 
            Session = {18606A09-966C-4049-884D-FCF602B38D5D}
 
            Job = {04D7CFF8-7808-4EC2-A5D0-EC73BBB52883}, "HKDC-BACKUP01 Backup 00218-Full"
 
            SessionFlags = 0x00000002
 
            JobFamilyID = {8A2C78AE-231F-41BA-8476-69AA3CDCB6DD}
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             + backupjob.cpp (5926):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             | BackupJob::MountThread : Device path hint :  m_deviceHost.empty() is TRUE
PVLSVR:   [07/19/12 11:39:41] [7356]     AdammSession::Execute( ADAMM_SESSION_EXECUTE_SESSION_SET_TARGET_DEVICE_HOST )
 
            Session = {18606A09-966C-4049-884D-FCF602B38D5D}
 
            Device Host = ""
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             + backupjob.cpp (5943):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             | BackupJob: acquired device 'Tape drive 0001'.
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             + backupjob.cpp (5966):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             | Setting device attributes 0x0
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             + backupjob.cpp (5971):
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             | BackupJob: trying to mount appending volume.
BENGINE:  [07/19/12 11:40:35] [1520]     2012-07-19T11:39:41.745 [server]             + storagedevice.cpp (332):
 
BE2012 BackupJob_20120719.PNG BE2012 ScratchMedia_20120719.PNG
Benjamin Chiu's picture

Funny, when I edit this backup to use all drives in VTL, I find that BE2012 only use the 6th or 7th drives only in the VTL!

I'm going crazy now...

Benjamin Chiu's picture

Yesterday, the Symantec support called me and tried to figure out what was happening. 

As troubleshoot with Symantec support, we finally deleted the VTL in data domain, restarted the services and tried to backup again. Sadly, the result still the same. Then he gathered the log for further investigation.

After disconnected the line with Symantec support, I found that he did not cancel the testing backup jobs before deleting the VTL in data domain. Hence, BE2012 reported an error message for some active jobs in the tape drives which were not existing.

And none of backup jobs can be started, all in "Queued" states. (I guessed the last 2 available tapes drives were occupied by the testing backup job.)

Unfortunately, that was after office hour!!!! crying I stayed in office over time for troubleshoot by myself...

And finally I had to logon to Database level to delete the orphan entries in "Device" table of BEDB database, and restarted the services.

Finally BE2012 now can use all the tape drives in VTL now. I hope this is the last time I call for Symantec....

SOLUTION