Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Problem setting up Netbackup SSO (Shared Storage Options)

Created: 23 Oct 2013 • Updated: 05 Nov 2013 | 19 comments
This issue has been solved. See solution.

Hi All,

 Our Netbackup system is version 7.5.0.4. We have 2 tape library (HP ESL322e) connect using SAN. We already had up running drives running as SSO (setup by our System Integrator). Two of  drives on the TL being shared between 2 server , one running Solaris 10 the other one AIX 6.1. in site A. There is another pair with the similar configuration on site B.

One day our HP did firmware upgrade to TL on site A. The upgrade was not smooth, I had to reconfigure again the connection on Solaris 10 (but the AIX was fine).

The problem if I setup individual tape drive to the Solaris  or AIX, the backup could run fine. Then I did the SSO with configuration like 1 only for solaris, another 1 only for AIX, but 2 other dirves I config as SSO. Then I test the backup on Solaris , it supposed can see 2 drives (1 single, 1SSO). I fired first backup it run on the Solaris, but when I run the second drive, it won' go that SSO drive, it keep on waiting for resources.

 

10/23/2013 16:17:16 - awaiting resource ESL025_EBS6_MPX. Waiting for resources.
          Reason: Drives are in use, Media server: ebs6-bck,
          Robot Type(Number): TLD(1), Media ID: N/A, Drive Name: N/A,
          Volume Pool: test1, Storage Unit: ESL025_EBS6_MPX, Drive Scan Host: N/A,
          Disk Pool: N/A, Disk Volume: N/A

notes: ebs6 is the Solaris media server.

 

What i did wrong? Why it cannot use that SSO drive?

 

Thank you and regards,

 

Iwan

Comments 19 CommentsJump to latest comment

RamNagalla's picture

what is the robot number for SSO Drive..?

when you have 2  tape Libraries.. you need to have 2 robots too..  one robot for tape library one and another for tape library 2.

each robot needs to have the seperate storage unit created and used..

so do you have 2 robots?

does it have 2 diffent storage units created?

 

 

Marianne's picture

How many drives configured in the Storage Unit?

Please show us STU config:

/usr/openv/netbackup/bin/admincmd/bpstulist -U

These commands as well on Solaris and AIX servers:

/usr/openv/volmgr/bin/tpconfig -l

/usr/openv/volmgr/bin/vmoprcmd -d

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Iwan Tamimi's picture

Hi all,

 

Thanks for the responses,

Nagalla: The two tape libraries are separated, they only shared the same master server, so actually I mentioned we have 2 tape library only for reference, the other tape library the SSO is still working fine. I am sorry to confuse yu.

Marianne:

It should go to the storage unit ESL025_EBS6_MPX (Solaris) and ESL025_EBS7_MPX (AIX)

Currently there is single drive (11)  for EBS6, single drive (14) and SSO drive on EBS6/7 (12) which is not working, 1 drive (13) is not configured yet. The robot connect to EBS3

ebs1moc1 ~]# /usr/openv/netbackup/bin/admincmd/bpstulist -U

Label:                ESL025_EBS6_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs6-bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        12
Density:              hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number:    TLD / 1
Max Fragment Size:    1048575 MB

Label:                ESL125_EBS13_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs13-bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        4
Density:              hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number:    TLD / 3
Max Fragment Size:    1048575 MB

Label:                ESL12_EBS11_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs11bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        10
Density:              hcart - 1/2 Inch Cartridge
Robot Type/Number:    TLD / 3
Max Fragment Size:    1048576 MB

Label:                ESL02_EBS3_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs3-bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        10
Density:              hcart - 1/2 Inch Cartridge
Robot Type/Number:    TLD / 1
Max Fragment Size:    1048576 MB

Label:                ESL02_EBS2_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs2-bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        12
Density:              hcart - 1/2 Inch Cartridge
Robot Type/Number:    TLD / 1
Max Fragment Size:    1048575 MB

Label:                ESL02_EBS5_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs5-bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        12
Density:              hcart - 1/2 Inch Cartridge
Robot Type/Number:    TLD / 1
Max Fragment Size:    1048576 MB

Label:                ebs11bck-4mm
Storage Unit Type:    Media Manager
Host Connection:      ebs11bck
Number of Drives:     1
On Demand Only:       no
Max MPX/drive:        1
Density:              4mm - 4mm Cartridge
Robot Type:           (not robotic)
Max Fragment Size:    1048576 MB

Label:                ebs3-bck-4mm
Storage Unit Type:    Media Manager
Host Connection:      ebs3-bck
Number of Drives:     1
On Demand Only:       no
Max MPX/drive:        1
Density:              4mm - 4mm Cartridge
Robot Type:           (not robotic)
Max Fragment Size:    1048576 MB

Label:                ESL025_EBS7_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs7-bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        12
Density:              hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number:    TLD / 1
Max Fragment Size:    1048575 MB

Label:                ESL125_EBS12_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs12-bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        12
Density:              hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number:    TLD / 3
Max Fragment Size:    1048575 MB

Label:                TEST_DD_VM
Storage Unit Type:    Disk
Storage Unit Subtype: DiskPool (6)
Host Connection:      ebs14bck
Concurrent Jobs:      80
On Demand Only:       yes
Max MPX:              1
Max Fragment Size:    524288 MB
Block Sharing:        no
Ok On Root:           no
Disk Pool:            ROC_DDBoost_DP
Snapshots:            no
Replication Primary:  no
Replication Source:   no
Replication Target:   no
Mirror            :   no
Independent       :   no

Label:                ROC_DDBoost_SU
Storage Unit Type:    Disk
Storage Unit Subtype: DiskPool (6)
Host Connection:      ebs14bck
Concurrent Jobs:      80
On Demand Only:       yes
Max MPX:              1
Max Fragment Size:    524288 MB
Block Sharing:        no
Ok On Root:           no
Disk Pool:            ROC_DDBoost_DP
Snapshots:            no
Replication Primary:  no
Replication Source:   no
Replication Target:   no
Mirror            :   no
Independent       :   no

Label:                REPL_DDMOC_SU
Storage Unit Type:    Disk
Storage Unit Subtype: DiskPool (6)
Host Connection:      _STU_NO_DEV_HOST_
Concurrent Jobs:      80
On Demand Only:       yes
Max MPX:              1
Max Fragment Size:    524288 MB
Block Sharing:        no
Ok On Root:           no
Disk Pool:            REPL_DDMOC_DP
Snapshots:            no
Replication Primary:  no
Replication Source:   no
Replication Target:   no
Mirror            :   no
Independent       :   no

Label:                EBS2_DISK_2
Storage Unit Type:    Disk
Storage Unit Subtype: Basic (1)
Host Connection:      ebs2-bck
Concurrent Jobs:      1
On Demand Only:       yes
Max MPX:              1
Path:                 "/usr/openv/netbackup/db/stu_disk_2"
Max Fragment Size:    1000 MB
Stage data:           no
Block Sharing:        no
High Water Mark:      98
Low Water Mark:       80
Ok On Root:           yes

Label:                EBS5_Disk
Storage Unit Type:    Disk
Storage Unit Subtype: Basic (1)
Host Connection:      ebs5-bck
Concurrent Jobs:      1
On Demand Only:       yes
Max MPX:              1
Path:                 "G:\Netbackup\DB\stu_disk"
Max Fragment Size:    4000 MB
Stage data:           no
Block Sharing:        no
High Water Mark:      98
Low Water Mark:       80
Ok On Root:           yes

Label:                EBS3_DISK
Storage Unit Type:    Disk
Storage Unit Subtype: Basic (1)
Host Connection:      ebs3-bck
Concurrent Jobs:      1
On Demand Only:       yes
Max MPX:              1
Path:                 "/usr/openv/netbackup/db/stu_disk"
Max Fragment Size:    2000 MB
Stage data:           no
Block Sharing:        no
High Water Mark:      100
Low Water Mark:       80
Ok On Root:           yes

Label:                EBS2_DISK
Storage Unit Type:    Disk
Storage Unit Subtype: Basic (1)
Host Connection:      ebs2-bck
Concurrent Jobs:      1
On Demand Only:       yes
Max MPX:              1
Path:                 "/usr/openv/netbackup/db/stu_disk"
Max Fragment Size:    1000 MB
Stage data:           no
Block Sharing:        no
High Water Mark:      100
Low Water Mark:       80
Ok On Root:           yes

Label:                EBS11_DISK
Storage Unit Type:    Disk
Storage Unit Subtype: Basic (1)
Host Connection:      ebs11bck
Concurrent Jobs:      1
On Demand Only:       yes
Max MPX:              1
Path:                 "/usr/openv/netbackup/db/stu_disk"
Max Fragment Size:    2000 MB
Stage data:           no
Block Sharing:        no
High Water Mark:      100
Low Water Mark:       80
Ok On Root:           yes

Label:                MOC_DDBoost_SU
Storage Unit Type:    Disk
Storage Unit Subtype: DiskPool (6)
Host Connection:      ebs5-bck
Concurrent Jobs:      80
On Demand Only:       yes
Max MPX:              1
Max Fragment Size:    524288 MB
Block Sharing:        no
Ok On Root:           no
Disk Pool:            MOC_DDBoost_DP
Snapshots:            no
Replication Primary:  no
Replication Source:   no
Replication Target:   no
Mirror            :   no
Independent       :   no

Solaris 10

root@ebs6-bck # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive       Robot                    Drive           Device         Second
Type     Num Index  Type DrNum Status  Comment    Name            Path           Device Path
robot      1    -    TLD    -       -  -          -               ebs3-bck
  drive    -    0 hcart2   11      UP  -          ESL025_Drive11  /dev/rmt/0cbn
robot      3    -    TLD    -       -  -          -               ebs11bck
  drive    -    1 hcart2   12      UP  -          ESL025_Drive12  /dev/rmt/1cbn
root@ebs6-bck # /usr/openv/volmgr/bin/vmoprcmd -d

                                PENDING REQUESTS

                                     <NONE>

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart2   TLD                -                     No       -         0
  1 hcart2   TLD                -                     No       -         0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 ESL025_Drive11        No       -
  1 ESL025_Drive12        Yes      -

AIX

ebs7:/ # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive       Robot                    Drive           Device
Type     Num Index  Type DrNum Status  Comment    Name            Path
robot      1    -    TLD    -       -  -          -               ebs3-bck
  drive    -    0 hcart2   14      UP  -          ESL025_Drive14  /dev/rmt3.1
robot      3    -    TLD    -       -  -          -               ebs11bck
  drive    -    1 hcart2   12      UP  -          ESL025_Drive12  /dev/rmt1.1
ebs7:/ # /usr/openv/volmgr/bin/vmoprcmd -d

                                PENDING REQUESTS

                                     <NONE>

                                  DRIVE STATUS

Drv Type   Control  User      Label  RecMID  ExtMID  Ready   Wr.Enbl.  ReqId
  0 hcart2   TLD                -                     No       -         0
  1 hcart2   TLD                -                     No       -         0

                             ADDITIONAL DRIVE STATUS

Drv DriveName            Shared    Assigned        Comment
  0 ESL025_Drive14        No       -
  1 ESL025_Drive12        Yes      -
ebs7:/ #

 

Regards,

 

Iwan Tamimi

Marianne's picture

Storage Unit is configured for 4 drives in robot 1:

Label:                ESL025_EBS6_MPX
Host Connection:      ebs6-bck
Number of Drives:     4
Robot Type/Number:    TLD / 1

 

While there are only 2 drives attached - 1 drive in robot 1 and another drive in robot 3:

root@ebs6-bck # /usr/openv/volmgr/bin/tpconfig -l

Device Robot Drive       Robot                    Drive           Device         Second
Type     Num Index  Type DrNum Status  Comment    Name            Path           Device Path
robot      1    -    TLD    -       -  -          -               ebs3-bck
  drive    -    0 hcart2   11      UP  -          ESL025_Drive11  /dev/rmt/0cbn 
robot      3    -    TLD    -       -  -          -               ebs11bck
  drive    -    1 hcart2   12      UP  -          ESL025_Drive12  /dev/rmt/1cbn

 

So, NBU will over-assign jobs for STU ESL025_EBS6_MPX that has in fact only one drive configured.

Is Drive12 in a different robot? There is no STU for this media server for robot 3.

If so, you need to create another STU and drop the number of drives in ESL025_EBS6_MPX to 1. Possibly a good idea to then group them in a STU group and select policies in the STUG. 

Same for AIX server - one drive in robot 1, one drive in robot 3.

Hope this helps.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Iwan Tamimi's picture

Marianne,

 

Aha, looks like that is the problem , the Drive 12 supposed to be in the robot 1 too. Hmm never notice that, thank you.  I remember I pointed to the robot 1, but let me check again.

Let me try to reconfigure it. Thanks for the early finding.

 

Regards,

Iwan Tamimi

Marianne's picture

Weird that you only have robot 1 and 3 configured. No 0 and 2?

Always best to use the Device Config wizard - it will read all drive serial numbers in the robot on the robot control host, and then config all drives in correct robot based on the serial numbers with correct drive position number.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Iwan Tamimi's picture

Marianne,

BTW I have changed it, but still cannot used the tape (I was testing on ebs6 and run 2 policies with different pull the firest one use the 11 but the second one just wait, not using the 12

Thu Oct 24 17:34:23 SGT 2013
root@ebs6-bck # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive       Robot                    Drive           Device         Second
Type     Num Index  Type DrNum Status  Comment    Name            Path           Device Path
robot      1    -    TLD    -       -  -          -               ebs3-bck
  drive    -    0 hcart2   11      UP  -          ESL025_Drive11  /dev/rmt/0cbn
  drive    -    1 hcart2   12      UP  -          ESL025_Drive12  /dev/rmt/1cbn
robot      3    -    TLD    -       -  -          -               ebs11bck

ebs7:/home/narahariak # /usr/openv/volmgr/bin/tpconfig -l
Device Robot Drive       Robot                    Drive           Device
Type     Num Index  Type DrNum Status  Comment    Name            Path
robot      1    -    TLD    -       -  -          -               ebs3-bck
  drive    -    0 hcart2   14      UP  -          ESL025_Drive14  /dev/rmt3.1
  drive    -    1 hcart2   12      UP  -          ESL025_Drive12  /dev/rmt1.1
robot      3    -    TLD    -       -  -          -               ebs11bck
ebs7:/home/narahariak #

 

Regards,

 

Iwan

 

Marianne's picture

Was NBU restarted on the media server?

It may also be a good idea to refresh config and the resource broker on the master server:

bprdreq -rereadconfig 

nbrbutil -resetAll 

 (only run last command when no backups are running).

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Iwan Tamimi's picture

I am so sorry for the very late reply due to some circumstances. 

I have rebooted the server and the media server but still failed for SSO but OK for single connection. 

 

Regards, 

Iwan

Marianne's picture

Please show us output for the following:

On master:
/usr/openv/netbackup/bin/admincmd/bpstulist -U
/usr/openv/netbackup/bin/admincmd/nbrbutil -dump
        (Just copy the MDS Allocation section at the bottom) 
/usr/openv/volmgr/bin/vmdareq
/usr/openv/volmgr/bin/vmoprcmd

Can I assume that 'tpconfig -l' on both servers are still as per your last post?

Is reason for queued jobs still 'Drives are in use'?

 

 

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Iwan Tamimi's picture

Marrianne, 

Thanks again for helping with the case.

I attached all the information you requested.

> Is reason for queued jobs still 'Drives are in use'?

Yes. on storage unit ESL025_EBS6_MPX consist of 1 single drive and 1 SSO drive, when the single drive empty then the job continue to that drive.

 

AttachmentSize
Netbackup_SSO_Problem.txt 20.61 KB
Marianne's picture

Thanks for the output.

Firstly, we see that stu ESL025_EBS6_MPX  is configured for 4 drives:

Label:                ESL025_EBS6_MPX
Storage Unit Type:    Media Manager
Host Connection:      ebs6-bck
Number of Drives:     4
On Demand Only:       no
Max MPX/drive:        12
Density:              hcart2 - 1/2 Inch Cartridge 2
Robot Type/Number:    TLD / 1
Max Fragment Size:    1048575 MB

Why is that when only 2 drives are assigned to this media server on this host?

As I have said before, this will result in over-allocation of jobs for this STU.

Can you confirm that SSO license key was added to both of these media servers? (Check output of get_license_key on both media servers).

vmoprcmd tells us that NBU was probably not restarted on media server ebs7 after drives were configured as SSO?

ESL025_Drive12           No      No                     No        hcart2
    ebs6-bck                   /dev/rmt/1cbn                        SCAN-TLD
    ebs7-bck                   /dev/rmt1.1                          RESTART

 

NBU needs to be restarted on both media servers.

Please correct the STU config on the master server and run 'bprdreq -rereadconfig'.

If above steps were taken and drive 12 is still not used, try to down drive 11 on ebs6 and see if drive 12 gets used.

 

Let us know the result.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

SOLUTION
RadovanTuran's picture

Try to play with SCSI reservation on tape drives. it can be configured globally and the each drive is using "Default" or you can change it per tape drive. SCSI-2 reservation doesn't work everytime with SSO but SCSI-3 reservation is not supported by all devices.

Play with both versions of config, check config of tape library. Many times firware upgrade change some values to defaults. Ask HP which SCSI reservation is supported by TL (SCSI-3 should be supported by new devices).

Iwan Tamimi's picture

Marrianne,

> Why is that when only 2 drives are assigned to this media server on this host?

Actually I wasn't sure why it was 4, now I undertand a bit,  I will change accordingly.

>Can you confirm that SSO license key was added to both of these media servers?

I think it should not be a problem, the configuration for both server have been in production for several month until the TapeLibrary upgrade problem. We didn't change any license config in both servers.

I will restart the NBU and give the ressutl.

 

Radovan,

The drives connected through fiber through the Fiber switch. The system actauly has been run for sometimes.

 

Regards, 

iwan

Marianne's picture

PLEASE double-check SSO licenses on both media servers.

We saw a similar issue here about a week ago when drive control remained in RESTART.
Issue was solved when SSO licenses were added to media servers.

Media servers should have these 3 licenses:
NBU Enterprise server (or Enterprise Client if SAN media server)
Library based tape drives
Shared Storage Option

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Iwan Tamimi's picture

Marianne,

Actually I also asked Symantec to help , and the support use the  "Configure Storage Devices" wizard on the Java GUI and after that seems ok. I didn't use the wizard when I configured it, I still don't know what I did wrong. Using the wizard , the support actually configure all the for drives to be SSO. After that I took out 1 drives from each media server (so each media server will get 3 drives which 2 of them are in SSO)then the problem come again.

Then I see your finding about my STU config, I corrected to 3 then looks the backup running fine.

Thank you very much.

Iwan

Marianne's picture

ALWAYS best to use the wizard with SSO config...

Glad it's working now!

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links