Video Screencast Help
Give us your opinion and win with Symantec! Please help us by taking this survey to tell us about your experience with Symantec Connect, so that we can continue to grow and improve.  Take the survey.

UNIX and Flashbackup

Created: 11 Dec 2012 | 24 comments

Background:

We had successfully tested Flashbackup using NetBackup 6.5.5 on UNIX media server. 

NBU Master - Windows 2003 using 6.5.5

NBU Media Server - UNIX HP-UX B.11.31 using 6.5.5

In the Policy it was setup:

Atrributes - Flashbackup

Backup Selections:

UNSET CACHE

CACHE=/dev/rdsk/c30t0d0

/dev/vgimagetest/rlvol1

Current Setup:

NBU Master - Windows 2003 using 7.5.1

NBU Media Server - UNIX HP-UX B.11.31 using 7.5.1

Backup Selections:

UNSET CACHE

CACHE=/dev/rdsk/c4t0d0

/dev/vgimage_test/rlvol1

Policy is the same.  When I run the job, it produces an error code of 12 (file open failed).  Also when I modify the policy, the system tries to validate the policy and I get this: Snapshot could not be created with VxFS_Snapshot method.  When I deselect "Perform snapshot backups" under Snapshot client, the validation is ok but I still get the same error.

Any thoughts of what is going on?

Comments 24 CommentsJump to latest comment

RamNagalla's picture

Backup Selections:

UNSET CACHE

CACHE=/dev/rdsk/c4t0d0

/dev/vgimage_test/rlvol1

 i belive /dev/vgimage_test/rlvol1  is not a Raw partiion .. you need to specify the Raw partion ..

See the below tech note.

http://www.symantec.com/business/support/index?page=content&id=HOWTO34898

copied from above T/N

For a FlashBackup policy, if the CACHE= entry follows the source data entry, the backup fails with status code 12. Messages such as the following appear in the/usr/openv/netbackup/logs/bpbkar logs on the client:

09:55:33.941 [6092] <16> bpfsmap: ERR - open_snapdisk: NBU 
snapshot enable failed error 3
09:55:33.942 [6092] <32> bpfsmap: FTL - bpfsmap: can't open 
snapshot disk /dev/rdsk/c4t1d0s3 errno 0
09:55:33.950 [6092] <16> bpbkar Exit: ERR - bpbkar FATAL exit 
status = 12: file open failed
09:55:33.956 [6092] <4> bpbkar Exit: INF - EXIT STATUS 12: file 
open failed
09:55:33.957 [6092] <2> bpbkar Exit: INF - Close of stdout 
complete

Change the order of the backup selections list so that the CACHE entry precedes the source data entry. (The source data entry specifies the raw partition that contains the file system to be backed up.)

hope this helps.

finalcloud's picture

Here is some more addition details:

12/11/2012 9:53:01 AM - Warning bpbrm(pid=17926) from client finux112: WRN - /dev/vgimage_test/rlvol1 is a character special file. Backing up the raw partition.
12/11/2012 9:53:01 AM - Error bpbrm(pid=17926) from client finux112: UX:vxfs mount: ERROR: V-3-25255: mount: You don't have a license to run this program

How can I verify the license?

Marianne's picture

It sounds as if the wrong snapshot option is selected in the policy.
Please show us all the information in NetBackup policy Attributes:

bpplinfo <policy-name> -L

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

finalcloud's picture

As requested.  Note I'm using the same policy I created when this worked for 6.5.5.

>bpplinfo z-test-finux112_WINS_images -L
Policy Type:            FlashBackup (20)
Active:                 yes
Effective:              02/16/2010 10:27:07
Follow NFS Mounts:      no
Cross Mount Points:     no
Client Compress:        no
Collect TIR info:       no
Policy Priority:        0
Ext Security Info:      no
File Restore Raw:       yes
Client Encrypt:         no
Max Jobs/Policy:        1
Mult. Data Stream:      yes
Block Level Incremental:        no
Perform Snapshot Backup:        no
Snapshot Method:                (none)
Snapshot Method Arguments:      (none)
Perform Offhost Backup:         no
Backup Copy:                    0
Use Data Mover:                 no
Data Mover Type:                2
Use Alternate Client:           no
Alternate Client Name:          (none)
Use Virtual Machine:            0
Hyper-V Server:         (none)
Enable Instant Recovery:        no
Collect BMR Info:       no
Max Frag Size:          0 MB
Checkpoint Restart:     no
Residence:              finux112-hcart-robot-tld-0
Volume Pool:            OneTime
Server Group:           *ANY*
Data Classification:            -
Residence is storage lifecycle policy:  no
Granular Restore:       no
Generation:     175
Ignore Client Direct:   no
Use Accelerator:        no
Exchange Source attributes:              no
Enable Metadata Indexing:       no
Application Discovery:  no
Discovery Lifetime:     0 seconds
ASC Application and attributes: (none defined)

Marianne's picture

I'm not close to any documentation right now, but you seem to be using the 'Legacy' Flashbackup policy config. Maybe this 'Legacy' method is no longer supported in 7.5?
Have you checked the 7.5 Snapshot Client manual to see if anything is new/different?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

finalcloud's picture

Going to uninstall and reinstall the client software.  Anything I should double check during that process?

Marianne's picture

Let's both have a look in the manual? http://www.symantec.com/docs/DOC5165

Maybe only policy needs to be updated.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Marianne's picture

p. 90 of the manual:

5 Select a snapshot method in one of the following ways:
■ Click Perform snapshot backups on the Attributes tab.
For a new policy, NetBackup selects a snapshot method when the backup starts.
For a copy of a policy that was configured for a snapshot method, click
the Snapshot Client Options option and set the method to auto. NetBackup
selects a snapshot method when the backup starts.
■ Click Perform snapshot backups, click the Snapshot Client Options option
and select a snapshot method.
See “Selecting the snapshot method” on page 64.
6 Windows only .....
7 UNIX only: if you selected nbu_snap or VxFS_Snapshot as the snapshot
method, specify a raw partition as cache, in either of these ways:
■ Use the Host Properties node of the Administration Console to specify
the default cache device path for snapshots. Click Host Properties >
Clients, select the client, then Actions>Properties, UNIXClient>Client
Settings.
■ Use the Snapshot Client Options dialog to specify the cache.
See “Entering the cache” on page 163.
The partition to be used for the cache must exist on all clients that are
included in the policy.
......
11 On the Backup Selections tab, specify the drive letter or mounted volume
(Windows) or the raw disk partition (UNIX) containing the files to back up.
 
HP examples:
/dev/rdsk/c1t0d0
/dev/vx/rdsk/volgrp1/vol1
 
Your 'backward compatility' method is still described in the manual.
Please try the 'newer' way as described above?
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

finalcloud's picture

Ok reinstalled the client piece on the UNIX server and made sure snapshot was licensed.  That issue appears to be resolved.  Created a new policy.

Policy Type: Flashbackup

Snapshot Client

Checked "Perform snapshot backups" clicked on "Options" -> Snapshot Method -> VxFS_Snapshot

Configuration Paramaters: Cache device path /dev/rdisk/disk14

I get a blank error message with the header "Netbackup Administration Console" I clicked "Yes"

Backup Selections: /dev/vgimage_test/rlvol1

Policy is validated

I get a new error:

12/11/2012 2:26:06 PM - Info bpbrm(pid=610) finux112 is the host to backup data from    
12/11/2012 2:26:06 PM - Info bpbrm(pid=610) reading file list from client       
12/11/2012 2:26:06 PM - Info bpbrm(pid=610) start bpfis on client        
12/11/2012 2:26:06 PM - begin Create Snapshot
12/11/2012 2:26:06 PM - Info bpfis(pid=613) Backup started          
12/11/2012 2:26:07 PM - snapshot backup of client finux112 using method VxFS_Snapshot
12/11/2012 2:26:07 PM - Critical bpbrm(pid=610) from client finux112: FTL - snapshot creation failed, status 156  
12/11/2012 2:26:07 PM - Warning bpbrm(pid=610) from client finux112: WRN - /dev/vgimage_test/rlvol1 is not frozen   
12/11/2012 2:26:07 PM - Info bpfis(pid=613) done. status: 156         
12/11/2012 2:26:07 PM - end Create Snapshot; elapsed time: 00:00:01
12/11/2012 2:26:07 PM - Info bpfis(pid=0) done. status: 156: snapshot error encountered      
12/11/2012 2:26:12 PM - begin Delete Snapshot
12/11/2012 2:26:12 PM - Info bpfis(pid=0) Snapshot will not be deleted       
12/11/2012 2:26:13 PM - Info bpfis(pid=625) Backup started          
12/11/2012 2:26:13 PM - Critical bpbrm(pid=622) from client finux112: FTL - cannot open /usr/openv/netbackup/online_util/fi_cntl/bpfis.fim.finux112_1355254021.1.0    
12/11/2012 2:26:13 PM - Info bpfis(pid=625) done. status: 1542         
12/11/2012 2:26:13 PM - end Delete Snapshot; elapsed time: 00:00:01
12/11/2012 2:26:13 PM - Info bpfis(pid=0) done. status: 1542: An existing snapshot is no longer valid and cannot be mounted for subsequent operations
12/11/2012 2:27:01 PM - Info nbjm(pid=6312) starting backup job (jobid=93600) for client finux112, policy z-test-FINUX112-Test, schedule OneTime 
12/11/2012 2:27:01 PM - Info nbjm(pid=6312) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=93600, request id:{44CEB608-03FD-4298-A6F6-3962FBDDD3A8}) 
12/11/2012 2:27:01 PM - requesting resource finux112-hcart-robot-tld-0
12/11/2012 2:27:01 PM - requesting resource finms-nbmaster.NBU_CLIENT.MAXJOBS.finux112
12/11/2012 2:27:01 PM - requesting resource finms-nbmaster.NBU_POLICY.MAXJOBS.z-test-FINUX112-Test
12/11/2012 2:27:01 PM - granted resource finms-nbmaster.NBU_CLIENT.MAXJOBS.finux112
12/11/2012 2:27:01 PM - granted resource finms-nbmaster.NBU_POLICY.MAXJOBS.z-test-FINUX112-Test
12/11/2012 2:27:01 PM - granted resource FC5917
12/11/2012 2:27:01 PM - granted resource IBM.ULTRIUM-TD4.063
12/11/2012 2:27:01 PM - granted resource finux112-hcart-robot-tld-0
12/11/2012 2:27:01 PM - estimated 0 Kbytes needed
12/11/2012 2:27:01 PM - begin Parent Job
12/11/2012 2:27:01 PM - begin Flash Backup, Start Notify Script
12/11/2012 2:27:01 PM - Info RUNCMD(pid=8464) started           
12/11/2012 2:27:01 PM - Info RUNCMD(pid=8464) exiting with status: 0        
Status 0
12/11/2012 2:27:01 PM - end Flash Backup, Start Notify Script; elapsed time: 00:00:00
12/11/2012 2:27:01 PM - begin Flash Backup, Step By Condition
Status 0
12/11/2012 2:27:01 PM - end Flash Backup, Step By Condition; elapsed time: 00:00:00
12/11/2012 2:27:01 PM - begin Flash Backup, Stream Discovery
Status 0
12/11/2012 2:27:01 PM - end Flash Backup, Stream Discovery; elapsed time: 00:00:00
12/11/2012 2:27:01 PM - begin Flash Backup, Read File List
Status 0
12/11/2012 2:27:01 PM - end Flash Backup, Read File List; elapsed time: 00:00:00
12/11/2012 2:27:01 PM - begin Flash Backup, Create Snapshot
12/11/2012 2:27:01 PM - started
12/11/2012 2:27:03 PM - started process bpbrm (610)
12/11/2012 2:27:06 PM - end writing
Status 156
12/11/2012 2:27:06 PM - end Flash Backup, Create Snapshot; elapsed time: 00:00:05
12/11/2012 2:27:06 PM - begin Flash Backup, Stop On Error
Status 0
12/11/2012 2:27:06 PM - end Flash Backup, Stop On Error; elapsed time: 00:00:00
12/11/2012 2:27:06 PM - begin Flash Backup, Delete Snapshot
12/11/2012 2:27:09 PM - started process bpbrm (622)
12/11/2012 2:27:13 PM - end writing
Status 1542
12/11/2012 2:27:13 PM - end Flash Backup, Delete Snapshot; elapsed time: 00:00:07
12/11/2012 2:27:13 PM - begin Flash Backup, End Notify Script
12/11/2012 2:27:13 PM - Info RUNCMD(pid=7156) started           
12/11/2012 2:27:13 PM - Info RUNCMD(pid=7156) exiting with status: 0        
Status 0
12/11/2012 2:27:13 PM - end Flash Backup, End Notify Script; elapsed time: 00:00:00
Status 156
12/11/2012 2:27:13 PM - end Parent Job; elapsed time: 00:00:12
snapshot error encountered(156)

Why Oh WHY did they put the Windows guy on this?!?

Marianne's picture

if I may ask - why all the different cache partitions?

NBU 6.5.5: CACHE=/dev/rdsk/c30t0d0

NBU7.5.0.1: CACHE=/dev/rdsk/c4t0d0
 
New policy: /dev/rdisk/disk14
(Is the 'i' a typo errror or do you really have this device name?)
 
We need bpfis log at this point in time....

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

finalcloud's picture

Please forgive the noob, bpfis log? 

Edit....Nevermind here is the log..

Edit 2.  After looking at this log, there is no mention of the policy and server I was using (FINUX112)

Edit 3....That log had nothing to do with this problem and has been removed.

Marianne's picture

Yep. Create bpfis and bpbkar directories on the client under /usr/openv/netbackup/logs.

New backup attempt will write log files (log.date) to one or both of these folders.
Snapshot attempt will be logged in bpfis.

Please copy logs to process name (e.g. bpfis.txt) and upload as File attachments.

Default logging level is 0. Most errors can be picked up with level 0 logs. If not, you will have to increase logging level by adding:
VERBOSE = 5
to /usr/openv/netbackup/bp.conf

(WARNING: I am allergic to level 5 logs... If needed, someone else will have to assist with analysis...)

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

finalcloud's picture

Please excuse the post above your, that was on the master server.  Attached is from the server in question.

AttachmentSize
FINUX112.121212.txt 6.6 KB
Marianne's picture

We need to find the reason for this error in bpfis log:

VxFS_Snapshot_mount: V_mount() failed with status=77, exit_status=1(Not owner)

(Will be back in a couple of hours...)

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

CRZ's picture

Hmmm, I'm more interested in this snippet:

09:54:29.911 [9225] <2> do_cmd:  argv[6]=/dev/rdisk/isk14

Is it really configured for /isk14?  Earlier up there you said /disk14.  (Actually, Marianne also asked if "rdisk" wasn't supposed to be "rdsk" ...so maybe TWO typos?)

I reckon you might have permission problems on a disk that didn't actually exist.  :) 

Confirm the path!  See if you can find that/those typo/typos and correct it/them, and see if that helps any.

If not, we'll probably need that same log again.


bit.ly/76LBN | APPLBN | 761LBN

Marianne's picture

I cannot find anything related to your error. The closest that i could find was: 

VxFS_Snapshot_mount: V_mount()failed with status=77, exit_status=16(Device busy)

But it is clearly not the same.

Have you in the meantime checked permissions and ownership on snapshot device as well as device to be backed up?

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

finalcloud's picture

I have requested that the UNIX guys take a look at the paths to answer the questions that were presented.  Thanks for all the help.

Marianne's picture

I was wondering about the same entry as Chris, but I also see this in the beginning of the log:

09:54:28.418 [9225] <2> logparams: bpfis create -nbu -owner NBU -id finux112_1355324121 -bpstart_to 300 -bpend_to 300 -backup_copy 0 -ru root -pt 20 -clnt finux112 -rg other -fim VxFS_Snapshot:cache=/dev/rdisk/disk14 -class z-test-FINUX112-Test -sched OneTime -fso -S finms-nbmaster -jobid 93768 
09:54:28.418 [9225] <2> bpfis main: received FIM as [37] VxFS_Snapshot:cache=/dev/rdisk/disk14

(Still wondering about 'rdisk' and why we see 3 different cache volumes across the thread)

So, it seems that 'do_cmd' is mis-interpreting arguments?

I would like to see results with one of the other cache volumes, e.g. /dev/rdsk/c4t0d0.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

finalcloud's picture

From the UNIX team

This is HP-UX 11.31, so the server has native multipathing.  You may want to post this:

some_filer:/vol/vol_images_cache_test/lun1 (LUN 1)

Agile Filename: /dev/rdisk/disk15       VG: /dev/vgimage_test

    50g (53687091200)    lun state: GOOD

Filer_CF_State: Cluster Enabled  Filer Partner: some_filer

Multipath_Policy: A/A

Multipath_Provider: HP A/A

-------- --------- -------------------- ------ ------- ------- -------------

host     filer                 /dev/dsk        primary partner HP A/A

path     path                  filename host   filer   filer   path failover

state    type          or Hardware Path HBA    port    port    priority

-------- --------- -------------------- ------ ------- ------- -------------

up       primary        /dev/dsk/c4t0d1 fcd0   0f              0

up       primary        /dev/dsk/c5t0d1 fcd0   0b              0

up       primary       /dev/dsk/c11t0d1 fcd2   0d              0

up       primary       /dev/dsk/c10t0d1 fcd2   0h              0

up       secondary      /dev/dsk/c6t0d1 fcd0           0b      1

up       secondary      /dev/dsk/c7t0d1 fcd0           0f      1

up       secondary     /dev/dsk/c12t0d1 fcd2           0h      1

up       secondary     /dev/dsk/c13t0d1 fcd2           0d      1

some_filer:/vol/vol_cache_test/lun0 (LUN 0)

Agile Filename: /dev/rdisk/disk14

    10g (10737418240)    lun state: GOOD

Filer_CF_State: Cluster Enabled  Filer Partner: some_filer

Multipath_Policy: None

Multipath_Provider: None

-------- --------- -------------------- ------ ------- --------

host     filer                                 primary partner

path     path                  /dev/dsk host   filer   filer

state    type                  filename HBA    port    port

-------- --------- -------------------- ------ ------- --------

up       primary        /dev/dsk/c4t0d0 fcd0   0f

up       primary        /dev/dsk/c5t0d0 fcd0   0b

up       secondary      /dev/dsk/c6t0d0 fcd0           0b

up       secondary      /dev/dsk/c7t0d0 fcd0           0f

up       primary       /dev/dsk/c11t0d0 fcd2   0d

up       primary       /dev/dsk/c10t0d0 fcd2   0h

up       secondary     /dev/dsk/c12t0d0 fcd2           0h

up       secondary     /dev/dsk/c13t0d0 fcd2           0d

The Agile Filename is the "disk" device that HP-UX creates so that when you use storage it can failover in the background.

Marianne's picture

I wonder if these multi-pathed disks are supported?

Maybe Chris can help here:

In Snapshot Client Admin Guide I see the following:

FlashBackup supports the following I/O system components: ufs, VxFS, and
Windows NTFS file systems, VxVM volumes and LVM volumes, and raw disks.
Other components (such as non-Veritas storage replicators or other non-Veritas
volume managers) are not supported.
 
When I look at the Snapshot Client CL, I cannot find FlashBackup support for HP-UX on anything other than VxVM volumes. 
(not the easiest doc to read/search...)

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

CRZ's picture

This is way over my head.  :) 

I guess one thing I'm confused about is your original post's CACHE variable was set in a way that specified NOT the multipathed....err, path... but one of its component paths, /dev/dsk/c4t0d0.

It looks like you changed it to /dev/rdisk/disk14 when you created a new policy from scratch?

There appear to be a lot of floating variables here - might be time to call support and open a case.  (They might very well tell you what Marianne concluded - that this isn't a supported configuration - in which case I wish I could tell you why it worked in 6.x but doesn't in 7.x, but unsupported configurations are usually unsupported with good reason)


bit.ly/76LBN | APPLBN | 761LBN

finalcloud's picture

Chris,

The server was rebuilt from scratch but it was on HP-UX 11.23 and needed to be on 11.31.  That might explain the different paths.  I will open ticket and reference this thread.

finalcloud's picture

Case #03211274 was created. 

Side note, I'm not a fan of the new method to submit tickets online. 

EDIT:

The case # is to create my access to view support cases...fail. 

Marianne's picture

Don't tell anyone, but I share your sentiments... I had the same experience a couple of weeks ago. Found out that it was the actual case #.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links