Video Screencast Help

NBU 7.5 - File system beeing backup up twice in the same schedule

Created: 24 Jan 2013 • Updated: 30 Jan 2013 | 23 comments
This issue has been solved. See solution.

Client: Centos 5.4 x86_64

Server: Solaris 10 Sparc

Both: NBU 7.5.0.4

Backup policy (standard):

  • ALL_LOCAL_DRIVES
  • Follow NFS / Cross Mountpoints are unchecked.
  • Allow multiple data strerams checked

On client:

  • /usr/openv/netbackup/include_list: /rawdata
  • /usr/openv/netbackup/exclude_list: /*

Problem:

NBU starts two jobs for the same file system /rawdata and finishes both jobs separately. There are several clients in the policy so I'll have to use the ALL_LOCAL_DRIVES directive...

 

Detaills - job 1:

24.jan.2013 06:00:22 - Info nbjm (pid=19595) starting backup job (jobid=48509) for client nrsk, policy UNIX-SERVERS, schedule FULL
24.jan.2013 06:00:22 - estimated 625900838 kbytes needed
24.jan.2013 06:00:22 - Info nbjm (pid=19595) started backup (backupid=nrsk_1359003622) job for client nrsk, policy UNIX-SERVERS, schedule FULL on storage unit hegre-hcart2-robot-tld-0
24.jan.2013 06:00:23 - started process bpbrm (pid=6595)
24.jan.2013 06:00:24 - Info bpbrm (pid=6595) nrsk is the host to backup data from
24.jan.2013 06:00:24 - Info bpbrm (pid=6595) reading file list from client
24.jan.2013 06:00:24 - connecting
24.jan.2013 06:00:25 - Info bpbrm (pid=6595) starting bpbkar on client
24.jan.2013 06:00:25 - Info bpbkar (pid=23169) Backup started
24.jan.2013 06:00:25 - Info bpbrm (pid=6595) bptm pid: 6615
24.jan.2013 06:00:25 - connected; connect time: 0:00:00
24.jan.2013 06:00:26 - Info bptm (pid=6615) start
24.jan.2013 06:00:27 - Info bptm (pid=6615) using 65536 data buffer size
24.jan.2013 06:00:27 - Info bptm (pid=6615) using 30 data buffers
24.jan.2013 06:00:27 - Info bptm (pid=6615) start backup
24.jan.2013 06:00:27 - Info bptm (pid=6615) backup child process is pid 6627
24.jan.2013 06:00:27 - Info bptm (pid=6615) Waiting for mount of media id A00018 (copy 1) on server hegre.
24.jan.2013 06:00:27 - mounting A00018
24.jan.2013 06:02:06 - Info bptm (pid=6615) media id A00018 mounted on drive index 1, drivepath /dev/rmt/4cbn, drivename IBM.ULTRIUM-HH5.001, copy 1
24.jan.2013 06:02:06 - mounted A00018; mount time: 0:01:39
24.jan.2013 06:02:06 - positioning A00018 to file 3
24.jan.2013 06:03:19 - positioned A00018; position time: 0:01:13
24.jan.2013 06:03:19 - begin writing
24.jan.2013 09:02:44 - Info bptm (pid=6615) waited for full buffer 482632 times, delayed 525891 times
24.jan.2013 09:02:52 - Info bptm (pid=6615) EXITING with status 0 <----------
24.jan.2013 09:02:52 - Info bpbrm (pid=6595) validating image for client nrsk
24.jan.2013 09:02:54 - Info bpbkar (pid=23169) done. status: 0: the requested operation was successfully completed
24.jan.2013 09:02:54 - end writing; write time: 2:59:35
the requested operation was successfully completed  (0)
 

Details - job 2:

24.jan.2013 06:00:23 - Info nbjm (pid=19595) starting backup job (jobid=48510) for client nrsk, policy UNIX-SERVERS, schedule FULL
24.jan.2013 06:00:23 - Info nbjm (pid=19595) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=48510, request id:{F5AC0C66-65E2-11E2-9A79-00212834954E})
24.jan.2013 06:00:23 - requesting resource Any
24.jan.2013 06:00:23 - requesting resource hegre.NBU_CLIENT.MAXJOBS.nrsk
24.jan.2013 06:00:23 - requesting resource hegre.NBU_POLICY.MAXJOBS.UNIX-SERVERS
24.jan.2013 06:00:23 - awaiting resource Any. No drives are available.
24.jan.2013 09:26:31 - granted resource  hegre.NBU_CLIENT.MAXJOBS.nrsk
24.jan.2013 09:26:31 - granted resource  hegre.NBU_POLICY.MAXJOBS.UNIX-SERVERS
24.jan.2013 09:26:31 - granted resource  A00068
24.jan.2013 09:26:31 - granted resource  IBM.ULTRIUM-HH5.000
24.jan.2013 09:26:31 - granted resource  hegre-hcart2-robot-tld-0
24.jan.2013 09:26:32 - estimated 626488281 kbytes needed
24.jan.2013 09:26:32 - Info nbjm (pid=19595) started backup (backupid=nrsk_1359015991) job for client nrsk, policy UNIX-SERVERS, schedule FULL on storage unit hegre-hcart2-robot-tld-0
24.jan.2013 09:26:33 - Info bpbrm (pid=15331) nrsk is the host to backup data from
24.jan.2013 09:26:33 - Info bpbrm (pid=15331) reading file list from client
24.jan.2013 09:26:33 - started process bpbrm (pid=15331)
24.jan.2013 09:26:33 - connecting
24.jan.2013 09:26:35 - Info bpbrm (pid=15331) starting bpbkar on client
24.jan.2013 09:26:35 - Info bpbkar (pid=25550) Backup started
24.jan.2013 09:26:35 - Info bpbrm (pid=15331) bptm pid: 15333
24.jan.2013 09:26:35 - connected; connect time: 0:00:00
24.jan.2013 09:26:36 - Info bptm (pid=15333) start
24.jan.2013 09:26:36 - Info bptm (pid=15333) using 65536 data buffer size
24.jan.2013 09:26:36 - Info bptm (pid=15333) using 30 data buffers
24.jan.2013 09:26:36 - Info bptm (pid=15333) start backup
24.jan.2013 09:26:36 - Info bptm (pid=15333) backup child process is pid 15336
24.jan.2013 09:26:36 - Info bptm (pid=15333) media id A00068 mounted on drive index 0, drivepath /dev/rmt/5cbn, drivename IBM.ULTRIUM-HH5.000, copy 1
24.jan.2013 09:26:36 - mounted A00068
24.jan.2013 09:26:36 - positioning A00068 to file 5
24.jan.2013 09:26:42 - positioned A00068; position time: 0:00:06
24.jan.2013 09:26:42 - begin writing
24.jan.2013 12:47:25 - Info bptm (pid=15333) waited for full buffer 573145 times, delayed 616927 times
24.jan.2013 12:47:32 - Info bptm (pid=15333) EXITING with status 0 <----------
24.jan.2013 12:47:33 - Info bpbrm (pid=15331) validating image for client nrsk
24.jan.2013 12:47:34 - Info bpbkar (pid=25550) done. status: 0: the requested operation was successfully completed
24.jan.2013 12:47:34 - end writing; write time: 3:20:52
the requested operation was successfully completed  (0)
 

Any ideas why??

BR,

 

Nils

 

Comments 23 CommentsJump to latest comment

Nagalla's picture

hi,

1) Do you have Multiple copies selected in schedule FULL  ?

How much of data each job is writing ? does it same size for both ?

 

does the storage unit is selected for policy or its set to Any Avaliable?

because i am seeing Any avaliable for 2nd job

24.jan.2013 06:00:23 - requesting resource Any

 

Nils K. Schøyen's picture

Hi,

Nope, Multiple copies not selected.

Attributes > Policy Storage unit is set to Any_available.

Amount of data is the same; the entire file system for both jobs....

 

BR,

Nils

 

Nagalla's picture

hi,

do the bpmount command on the client and see how may time the file systems are showing there

please provide the output of bpmount command form client.

William Jansen van Nieuwenhuizen's picture

hi

What happens if you open the BAR gui. It should show two id's, if you select them from within the bar gui, do they not show in the browselist how the data is different or the same?

Nils K. Schøyen's picture

[root@nrsk ~]# /usr/openv/netbackup/bin/bpmount
ext3: /dev/cciss/c0d0p1 on /
PROC: proc on /proc
sysfs: sysfs on /sys
devpts: devpts on /dev/pts
tmp: tmpfs on /dev/shm
ext3: /dev/cciss/c0d1p1 on /nrs
ext3: /dev/cciss/c0d1p2 on /rawdata
binfmt_misc: none on /proc/sys/fs/binfmt_misc
rpc_pipefs: sunrpc on /var/lib/nfs/rpc_pipefs
EXIT STATUS 0: the requested operation was successfully completed

 

In the BAR GUI I see two different jobs with different jobIDs. See details in first post.
 

BR,

 

Nils

Nicolai's picture

http://www.symantec.com/docs/TECH31513

Use this procedure to setup bpbkar logging. Once both backup has run inspect the log to see how the exclude/include list are being pressed.

Setting VERBOSE = 5 is required. 

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Nicolai's picture

Do you have /rawdata and ALL_LOCAL_DRIVES specified in the same file list (don't think so, but I need to ask) ?

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Nils K. Schøyen's picture

No, backup selection contains only one entry: ALL_LOCAL_DRIVES.

I have changed the exclude_list from '/*' to '*'.

My backup window does not permit new backups until tuesday, I'll try with bpbkar logging then.

 

BR,

 

Nils

 

Nils K. Schøyen's picture

I removed /usr/openv/netbackup/db/images/nrsk/STREAMS* and restarted NBU on the server.

Started a manual, FULL backup of the client. I see two jobs in the NBU Java GUI; one for / and one for /rawdata (shown in Job overview > File list). However, both jobs are taking backup of /rawdata...

# /usr/openv/netbackup/bin/bpmount
ext3: /dev/cciss/c0d0p1 on /
PROC: proc on /proc
sysfs: sysfs on /sys
devpts: devpts on /dev/pts
tmp: tmpfs on /dev/shm
ext3: /dev/cciss/c0d1p1 on /nrs
ext3: /dev/cciss/c0d1p2 on /rawdata
binfmt_misc: none on /proc/sys/fs/binfmt_misc
rpc_pipefs: sunrpc on /var/lib/nfs/rpc_pipefs
EXIT STATUS 0: the requested operation was successfully completed
 

 

Have not enabled bpbkar-logging yes as I was hoping removing the STREAMS-files would do the trick...

Nicolai's picture

Have you tried to addd policy definition to the exclude list ?

E.g. Let's say you have UNIX file system backup policy UNIX_FS.  You would then name the exlcude list

/usr/openv/netbackup/exclude_list.UNIX_FS

and

/usr/openv/netbackup/include_list.UNIX_FS

 

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Nils K. Schøyen's picture

Yes, I tried that on the server. No impact.

I'm attaching the bpbkar-logs from the client. The jobs are not finished but as you can see from the log, two jobs were started.

Ideas anyone?

AttachmentSize
log.012813.gz 1.47 KB
Nicolai's picture

I need more from the bpbkar log. I can't see bpbkar evaluating the exclude list.

It look somthing like :

10:53:16.574 [24681] <4> is_excluded: Excluded /oracledata/SID.dbf by exclude_list entry *.dbf

 

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Nicolai's picture

Please also verify that the Netbackup version on the client really is 7.5.0.4

Netbackup client 7.0 through 7.1.0.1 ignoring the exclude list

http://www.symantec.com/docs/TECH150101

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Nagalla's picture

hi,

Bpbkar log does not looks like with VERBOSE 5,

please provide the bpbkar log with VERBOSE 5

Nils K. Schøyen's picture

[root@nrsk bin]# cat /usr/openv/netbackup/bin/version
NetBackup-RedHat2.6.18 7.5.0.4
 

# cat /usr/openv/netbackup/bp.conf
SERVER = hegre
CLIENT_NAME = nrsk
SERVER_SENDS_MAIL = YES
VERBOSE = 5
 

???

Nicolai's picture

Good - we can now exclude both.

When can you upload a complete bpbkar log ?.

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

Yasuhisa Ishikawa's picture

It just looks like include_list affect both child jobs.

Why don't you configure dedicated policy for /rawbackup? Why you use such complex configuration?

 

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan

Nils K. Schøyen's picture

The configuration is straight-forward when dealing with policies with multiple clients.

bpbkar logfile included.

AttachmentSize
log.012813.gz 1.79 KB
Will Restore's picture

The trouble is  Allow multiple data streams  is selected in the Policy so the job starts with two streams and then each stream goes through the include_list/exclude_list on the client and both streams end up pointing to the same filesystem (/rawdata). 

 

Will Restore -- where there is a Will there is a way

SOLUTION
revaroo's picture

Why the include list? Just stick the list of backup selections in the policy.

Will Restore's picture

I believe he alluded to many clients in this single policy. 

Sure, it seems simple until it doesn't work.  Search this forum on "exclude  multiple data streams" for many instances of folks trying and not getting it to work how they like.

 

Putting the client in its own policy with straightforward Backup Selection is the answer. 

Or turn off Allow multiple data streams. 

Will Restore -- where there is a Will there is a way

Nils K. Schøyen's picture

I have tried the following on my NBU server (hegre):

hegre# /usr/openv/netbackup/bin/admincmd/bpclient  -client nrsk -L
Client Name: nrsk
 Current Host:
        Hostname: nrsk
 Dynamic Address:       no
 Free Browse:   Allow
 List Restore:  Not Specified
 Max Jobs This Client:  Not Specified
 WOFB Enabled:  yes
 WOFB FIM:      VSS
 WOFB Usage:    Individual Drive Snapshot
 WOFB Error Control:    Abort on Error
 Client Direct: Deduplication on the media server or
                Move data via media server
 Client Direct Restore: Move data via media server
 OST Proxy:     Off
 OST Proxy Server:      Unspecified
Connect options:        2 2 3
 Offline:       No

hegre# /usr/openv/netbackup/bin/admincmd/bpclient  -client nrsk -update  -max_jobs 1

hegre# /usr/openv/netbackup/bin/admincmd/bpclient  -client nrsk -L
Client Name: nrsk
 Current Host:
        Hostname: nrsk
 Dynamic Address:       no
 Free Browse:   Allow
 List Restore:  Not Specified
 Max Jobs This Client:  1
 WOFB Enabled:  yes
 WOFB FIM:      VSS
 WOFB Usage:    Individual Drive Snapshot
 WOFB Error Control:    Abort on Error
 Client Direct: Deduplication on the media server or
                Move data via media server
 Client Direct Restore: Move data via media server
 OST Proxy:     Off