Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

persistent tape drive bindings on Linux, multiple paths, etc.

Created: 09 Apr 2014 • Updated: 09 Apr 2014 | 15 comments

Hi,

We have had LTO5 tape drives zoned into our RHEL media server for some time. It appears that the /dev/nstx paths change whenever the box is rebooted and causes confusion more than anything, NBU seems to handle these path changes, but is there a way to prevent them from moving around? I've noticed that sometimes our drive labled "drive 2" occasionally picks up the physical path of some other drive so that the logical representation in device manager is just that, it doesn't always stay the same. We are not doing SSO, but plan to.

The other thing which I find weird but unexplainable, is that 2 of the drives show up with 3 "paths" when seen with the scan util, but the 2 most recently added drives only have 2 (which is what we expect since each tape drive is present on both fabrics but only one zone (one HBA per fabric).

# /usr/openv/volmgr/bin/scan |grep -c 101200328E

6

/usr/openv/volmgr/bin/scan |grep -c 101100328E

6

# /usr/openv/volmgr/bin/scan |grep -c 101300328E

4

# /usr/openv/volmgr/bin/scan |grep -c 101400328E

4

# /usr/openv/volmgr/bin/scan |grep -c 101400328E

 

thanks

Operating Systems:

Comments 15 CommentsJump to latest comment

Marianne's picture

Persistent Binding is always a good idea.

Download the hba tool for your hba (e.g SanSurfer for Qlogic, HbAnyware for Emulex) to assist with Persistent Binding.

Support site for your hba will give you details.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Ah45's picture

Also Enable automatic path correction

Enabling automatic path correction on Windows or LInux its same
You can configure NetBackup to automatic device path correction. To do so, use the following procedure.

To configure automatic path correction

Use a text editor to open the following file:

install_path\VERITAS\Volmgr\vm.conf

Add the following AUTO_PATH_CORRECTION entry to the file:

AUTO_PATH_CORRECTION = YES

If it already exists but is set to NO, change the value to YES.

Save the file and exit the text editor.

 

follow the link

http://www.symantec.com/business/support/index?pag...

schrammd's picture

auto path correction is set to yes and presumably this is why it generally keeps working when the underlying paths are changing. The persistent binding thing I am investigating, but trying to set this via hbacmd states that "HBACMD_GetPersistentBinding: Persistent Binding configuration is not available"

thanks

mph999's picture

The HBA (hopefully Emulex or Qlogic) will have a config utiiity unique to the brand, Sansurfer (Qlogic) and HBAnywere (Emulex)  (hopefully I got those the right way round ...)

Via these GUIs usually allows the easiest way to set up the persistent binding.

Given that this is a NBU forum, if you have issues it may be qquickest for you to contact the appropriate vendor.

 

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
schrammd's picture

Fact is, the persistent binding commands are only applicable to Windoze. So, any other suggestions?

Basically what I see is there is some confusion on the part of the built-in NBU multipathing algorithm (I'm assuming this is what is taking care of it since DM-multipath is not), so, how can I flush out all the old/redundant path refrences and make the scan util actuall show me the real paths in use for each tape device?

Maybe I'm missing something but I've consulted the hbacmd manual and this is not applicable to Linux.

thx

 

mph999's picture

As mentioned, the GUI utility should work, across all platforms.

Regarding the NBU issue, you say old redudant paths - are you seeing MISSING PATH in tpconfig -d output by any chance ?

Multipathing is separate from the auto re config, but that's not to say there isn't something odd going on, but basically NBU should re-jig the paths and then delete the old one, but, under certain conditions I think it can leave a missing path behind.  Unfortunately, I've not worked out exactly under what conditions this happens - I'm not a programmer, and hence my ability to follow through the code is a bit painful.

Hmm, just re-read your last post ...

"... make the scan util actually show me the real paths ..."

There is no control over the scan command, I apreciate this is a Symantec supplied command but it is 'seperate' from NBU, as in it doesn't use NBU in any way (it will run with NBU stopped) - apart from a bit of fomatting, it only returns results of devices seen by the OS.

Could you perhaps post up the complete scan output and highlight the issue so we can be sure we understand correctly.

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
schrammd's picture

yeah, sorry, too busy shaking things to try and see what will clear this up. I have some idea that possibly there is a bug in the Brocade code which has left ghost ports/zones in the fabric even though the set appears to be what we want. The more detailed explanation is that we have had issues over the last year or so with the /dev/nst* devices moving around under the "name" we give them as devices. We recently added more fiber paths to the box in order to seperate the disk traffic from the tape. So, 2 of these drives previously were zoned into the "old" cards and they now are zoned into the new ones. Drive 1 and 2 are reporting more paths than is physically possible, at least from the zoneset in place. Anyhow, as of now, the HBA drivers are indeed reflecting what it thinks is zoned in and NBU is just adding those old paths. It smells of a ghost zone or 2 in the fabric. Drive 3 and 4 are OK except for me trying to understand how to get the /dev/nst paths to not move around, but if NBU can just take care of it automatically, then I'll move on to just accepting it as is.

Id DriveName Type Residence

Drive Path Status

****************************************************************************

0 BACKUP104-LTO5-D1 hcart3 TLD(0) DRIVE=1

/dev/nst0 UP

/dev/nst8 UP

/dev/nst5 UP

1 BACKUP104-LTO5-D2 hcart3 TLD(0) DRIVE=2

/dev/nst4 UP

/dev/nst9 UP

/dev/nst2 UP

2 BACKUP104-LTO5-D3 hcart3 TLD(0) DRIVE=3

/dev/nst7 UP

/dev/nst3 UP

3 BACKUP104-LTO5-D4 hcart3 TLD(0) DRIVE=4

/dev/nst1 UP

/dev/nst6 UP

 

# /usr/openv/volmgr/bin/scan

************************************************************

*********************** SDT_TAPE ************************

*********************** SDT_CHANGER ************************

************************************************************

------------------------------------------------------------

Device Name : "/dev/nst8"

Passthru Name: "/dev/sg248"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101100328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101100328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst9"

Passthru Name: "/dev/sg249"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101200328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101200328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst7"

Passthru Name: "/dev/sg55"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101300328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101300328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst6"

Passthru Name: "/dev/sg54"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101400328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101400328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst5"

Passthru Name: "/dev/sg53"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101100328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101100328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst4"

Passthru Name: "/dev/sg52"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101200328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101200328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst2"

Passthru Name: "/dev/sg45"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101200328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101200328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst3"

Passthru Name: "/dev/sg46"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101300328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101300328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst0"

Passthru Name: "/dev/sg43"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101100328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101100328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst1"

Passthru Name: "/dev/sg44"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101400328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101400328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

 

 

mph999's picture

You explanation sounds good, if it's the actual cause I don;t know as I've never seen this before.

What is confusing is say for drive 

Device Name : "/dev/nst8"

Passthru Name: "/dev/sg248"

 

All the other paths report the inquiry string, so far as I know, contact was made and a response sent back.

You could prove this ...

/usr/openv/volmgr/bin/scsi_command -d <path>

Eg.

root@nbmaster00 bpdbm $ /usr/openv/volmgr/bin/scsi_command -d /dev/nst0
Inquiry data: removable dev type 1h IBM     ULT3580-TD4     550V

root@nbmaster00 bpdbm $ /usr/openv/volmgr/bin/scsi_command -d /dev/sg3
Inquiry data: removable dev type 1h IBM     ULT3580-TD4     550V

Seems to can use either path ...

 

Would be good to confirm that all three paths for one of the problem drive respond (they will .. ;0))

Regards,

Martin

 

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
schrammd's picture

Ok, update. We did in fact have a problem in the fabric (one of the 2) whereby the zones from long ago were haunting the current zoneset. We have fixed that and pruned up the fabrics. Now, the real rub is that the (lack of) persistent binding is a PITA. As stated, Emulex does not support forcing bindings on these cards with Linux. So, I opened a case with RedHat support and their people have been outstanding helping us get the udev rules figured out. Problem is and has been last time we tried this on linux is that NBU cannot interpret the alias tape drive names even though from the OS perspective they are fully 100% valid. If I add the drive into NBU, 2 things become very obvious - the SN of the drive is unknown and the second path to the same drive is no longer visible. Try to run a job to the drive, no dice. Cannot control the drive using any alias. We even tried using the /dev/tape/by-id path, same result. So, back to Symantec again - why is this is and how do we get NBU to use persistent bindings in RHEL? The issue is and has been rebooting the box shakes up the bindings and multipathing begins to breakdown.

thanks,

 

Dave

RonCaplinger's picture

http://www-dl.emulex.com/support/linux/732/set.pdf

Page 68, "Add Persistent Binding"

If you already checked this out, I apologize, I didn't compare versions of drivers/OS's to yours.

From my history with this feature, HBA Persistent Binding is a function between the OS and the HBA.  NBU doesn't use Persistent Binding; the OS presents the paths to NBU.  If you are unable to get the HBA to use persistent binding, there's nothing NBU can do about it. See if you can find some Emulex forums that might give some help with PB in Linux.

Marianne's picture

Excellent post from Ron.

All that NBU is 'concerned' about is the output from scsi-commands sent to the tape drives.

You can see how the OS is responding to this request with /usr/openv/volmgr/bin/scan command.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

schrammd's picture

That doc is way outdated. From Emulex's recent driver doc:

Note: Persistent binding is not supported by the Linux 2.6 kernel, the Emulex 8.2 version of the driver for Linux or by VMware ESX Server.

Seems like there is not much cohesion between the Red Hat folks, the Emulex people, and Symantec these days. As far as what NetBackup sees for the drive added to NBU devices using the udev-rule-created-alias of "/dev/nT950_Drive4", scan still picks up the /dev/nst paths. Are you suggesting that we cannot use udev rules to setup persistent bindings even though this IS the recommended way from RH? That seems a bit "off" to us.

# /usr/openv/volmgr/bin/scan

************************************************************

*********************** SDT_TAPE ************************

*********************** SDT_CHANGER ************************

************************************************************

------------------------------------------------------------

 

------------------------------------------------------------

Device Name : "/dev/nst5"

Passthru Name: "/dev/sg22"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101400328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101400328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

Device Name : "/dev/nst1"

Passthru Name: "/dev/sg3"

Volume Header: ""

Port: -1; Bus: -1; Target: -1; LUN: -1

Inquiry : "IBM ULTRIUM-TD5 D8D4"

Vendor ID : "IBM "

Product ID : "ULTRIUM-TD5 "

Product Rev: "D8D4"

Serial Number: "101400328E"

WWN : ""

WWN Id Type : 0

Device Identifier: "IBM ULTRIUM-TD5 101400328E"

Device Type : SDT_TAPE

NetBackup Drive Type: 10

Removable : Yes

Device Supports: SCSI-6

Flags : 0x0

Reason: 0x0

------------------------------------------------------------

 

Basically I can name the drive whatever using udev rules but NBU isn't smart enough to read the device from there.

Please, explain why the OS methodology is not good enough for the product....

# sg_inq /dev/nT950_Drive4

standard INQUIRY:

PQual=0 Device_type=1 RMB=1 version=0x06 [SPC-4]

[AERC=0] [TrmTsk=0] NormACA=0 HiSUP=0 Resp_data_format=2

SCCS=0 ACC=0 TPGS=0 3PC=0 Protect=1 BQue=0

EncServ=0 MultiP=0 [MChngr=0] [ACKREQQ=0] Addr16=0

[RelAdr=0] WBus16=0 Sync=0 Linked=0 [TranDis=0] CmdQue=1

[SPI: Clocking=0x0 QAS=0 IUS=0]

length=70 (0x46) Peripheral device type: tape

Vendor identification: IBM

Product identification: ULTRIUM-TD5

Product revision level: D8D4

Unit serial number: 101400328E

 

To them, as us, this is an application issue, not an HBA/driver/OS issue.

thanks

Marianne's picture

Correct. NBU uses /dev/nstx device names.

Please see Linux chapter of Device Config Guide: http://www.symantec.com/docs/DOC3656

 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

schrammd's picture

Ah, yes. This is helpful indeed. Apparently upon building we missed the part that says if doing multipathing you must use the /dev/nstx devices, thus, this is why the udev rule aliases fail to work. Still, I think this is balogna, and it does nothing to solve the problem of the paths changing at reboot time. So, since Emulex cannot do the bindings the old way under the newer kernels, and we cannot use the OS tools to force the binding through udev, and your application is fixed on mounting drives through /dev/nstx naming, then what? What IS the solution? Is it possible that if we remove the multipathing to all drives we would have less(meaning no issue) of an issue? Seems the solution is nearby, or not, but I don't want to remove the multipathing unless it simply cannot work with it. Would changing to Qlogic HBAs resolve the issue? There is nothing else in the manual that addresses this question that I can find.

 

thanks

 

Michael G Andersen's picture

A possible solution could be make to a boot script that removed tape device files and then added them in the order you want. Have used this approach for an AIX system some years ago.

If you have not be through it already the emulex's "An Insider's Guide to the Linux 2.6 Kernel" is probably a look worth.