Video Screencast Help

db_FLISTsend failed: system call failed (11)

Created: 29 Apr 2013 | 9 comments

I installed Netbackup 7.5.0.4 in the beginning of December, 2012 to backup a NetApp via NDMP. The problem I am now experiencing is that the image directory in the openv/netbackup/db/ is pegged at 100% full. That partition is 146GB. There are aprox. 160 .lck files under this directory dating back to December. Does anyone have suggestions as to how to proceed? What might the best way to trim the images directory be? Why are there so many .lck files hanging around? Is there a way to minimize the size of the catalog files?

Thank you,

DR

Operating Systems:

Comments 9 CommentsJump to latest comment

Nicolai's picture

Extend the file system. If you are on Solaris this should be a stright forward operation.

The only way to reduce the size of the catalog is to 

  • expire backups (not recommended)
  • compress the catalog (temporary pain relief) see - http://www.symantec.com/docs/HOWTO33397

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

drhartman's picture

Thank you, Nicolai. But does this mean that in one year I could need almost a half a terabyte worth of disk space just for the catalogs? Is there a way to reduce the size of the catalogs to begin with? (many of the files are greater than 10GB) I don't recall having this issue in the past with earlier versions of Netbackup...also, why are there so many .lck files still laying around? I thought that these got removed once the backup policy for that day was finished?

Thanks,

DR 

Marianne's picture

Image size is determined by number of files backed up on clients and retention levels.

You may want to go through these topics in chapter 17 of NBU Admin Guide I:

 

Protecting the NetBackup catalog
 
■ About the NetBackup catalog
■ About the catalog upon upgrade to NetBackup 7.5
■ Parts of the NetBackup catalog
■ Protecting the NetBackup catalog
■ Recovering the catalog
■ Disaster recovery emails and the disaster recovery file
■ Archiving the catalog
Estimating catalog space requirements
 

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Nicolai's picture

Without knowing that I am almost 100% sure the .lck files is not related to active backups.

# find . -name ".lck"

./1366000000/tmp/.lck
./1366000000/.lck
./1367000000/tmp/.lck
./1367000000/.lck
./1365000000/tmp/.lck
./1365000000/.lck
./1364000000/tmp/.lck
./1364000000/.lck
 
It looks like there is a .lck file for each directory in time
 
# bpdbm -ctime 1366000000
1366000000 = Mon Apr 15 06:26:40 2013
# bpdbm -ctime  1367000000
1367000000 = Fri Apr 26 20:13:20 2013
]# bpdbm -ctime  1365000000
1365000000 = Wed Apr  3 16:40:00 2013
# bpdbm -ctime  1364000000
1364000000 = Sat Mar 23 01:53:20 2013

 

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

drhartman's picture

Thank you very much...once again...Perhaps you know of the reason why I have so many more .lck files and in different locations...Please keep in mind that there have been many backup failures since the initial install of Netbackup 7.5. (more failures than successes). I am not sure how many are even valid.

Here is the output from a similar find command: 

** There are 2 policies that define everything necessary to be backed up on the filer on a regular basis...NetAppServer and NetAppServer_VOBS. NetAppServer has 5 volumes to back up and NetAppServer_VOBS has 3 volumes to back up.

NetbackupServer# pwd
/opt/openv/netbackup/db/images/NetAppServer
NetbackupServer# ls
1348000000                 1355000000                 1364000000                 STREAMSNetAppServer_VOBS.lck
1349000000                 1357000000                 1365000000                 STREAMSNetAppServer.lck
1350000000                 1360000000                 1366000000                 STREAMSTest
1351000000                 1361000000                 1367000000                 STREAMSTest.lck
1352000000                 1362000000                 STREAMSNetAppServer           STREAMStest2
1354000000                 1363000000                 STREAMSNetAppServer_VOBS      STREAMStest2.lck
NetbackupServer# find . -name \*.lck
./1348000000/.lck
./1348000000/NetAppServer_1348885096_FULL.lck
./STREAMSNetAppServer_VOBS.lck
./STREAMSNetAppServer.lck
./1349000000/.lck
./1349000000/NetAppServer_VOBS_1349695320_FULL.lck
./1350000000/NetAppServer_1350691202_FULL.lck
./1350000000/.lck
./1350000000/NetAppServer_1350691800_FULL.lck
./1350000000/NetAppServer_1350694497_FULL.lck
./1350000000/NetAppServer_VOBS_1350708108_FULL.lck
./1350000000/NetAppServer_VOBS_1350708106_FULL.lck
./1351000000/.lck
./1351000000/NetAppServer_VOBS_1351915203_FULL.lck
./1351000000/NetAppServer_VOBS_1351982598_FULL.lck
./1351000000/NetAppServer_VOBS_1351915201_FULL.lck
./1352000000/NetAppServer_VOBS_1352035888_FULL.lck
./1352000000/.lck
./STREAMSTest.lck
./STREAMStest2.lck
./1362000000/.lck
./1362000000/NetAppServer_1362324485_FULL.lck
./1362000000/NetAppServer_1362324483_FULL.lck
./1362000000/NetAppServer_VOBS_1362805206_FULL.lck
./1362000000/NetAppServer_VOBS_1362827852_FULL.lck
./1362000000/NetAppServer_1362961380_FULL.lck
./1354000000/.lck
./1354000000/NetAppServer_1354430398_FULL.lck
./1354000000/NetAppServer_VOBS_1354397080_FULL.lck
./1354000000/NetAppServer_VOBS_1354401572_FULL.lck
./1354000000/NetAppServer_1354500000_FULL.lck
./1354000000/NetAppServer_1354942806_FULL.lck
./1354000000/NetAppServer_VOBS_1354942804_FULL.lck
./1355000000/NetAppServer_1355022666_FULL.lck
./1355000000/NetAppServer_1355069533_FULL.lck
./1355000000/.lck
./1360000000/.lck
./1360000000/NetAppServer_1360386006_FULL.lck
./1360000000/NetAppServer_VOBS_1360386572_FULL.lck
./1360000000/NetAppServer_VOBS_1360386004_FULL.lck
./1360000000/NetAppServer_VOBS_1360393658_FULL.lck
./1360000000/NetAppServer_1360435815_FULL.lck
./1360000000/NetAppServer_1360447397_FULL.lck
./1360000000/NetAppServer_VOBS_1360386001_FULL.lck
./1360000000/NetAppServer_1360465613_FULL.lck
./1360000000/NetAppServer_VOBS_1360990806_FULL.lck
./1357000000/NetAppServer_1357362006_FULL.lck
./1357000000/NetAppServer_1357476034_FULL.lck
./1357000000/.lck
./1357000000/NetAppServer_VOBS_1357362004_FULL.lck
./1357000000/NetAppServer_VOBS_1357427140_FULL.lck
./1357000000/NetAppServer_VOBS_1357470515_FULL.lck
./1357000000/NetAppServer_VOBS_1357362001_FULL.lck
./1357000000/NetAppServer_1357494285_FULL.lck
./1357000000/NetAppServer_1357494425_FULL.lck
./1357000000/NetAppServer_1357499251_FULL.lck
./1357000000/NetAppServer_1357362003_FULL.lck
./1367000000/NetAppServer_1367035203_FULL.lck
./1367000000/NetAppServer_VOBS_1367035206_FULL.lck
./1367000000/NetAppServer_VOBS_1367035907_FULL.lck
./1367000000/NetAppServer_1367119564_FULL.lck
./1367000000/NetAppServer_VOBS_1367133348_FULL.lck
./1367000000/NetAppServer_1367179209_FULL.lck
./1367000000/NetAppServer_VOBS_1367183199_FULL.lck
./1367000000/NetAppServer_1367185394_FULL.lck
./1367000000/NetAppServer_VOBS_1367196357_FULL.lck
./1367000000/NetAppServer_1367210690_FULL.lck
./1367000000/.lck
./1367000000/NetAppServer_1367230962_FULL.lck
./1361000000/.lck
./1361000000/NetAppServer_1361595604_FULL.lck
./1361000000/NetAppServer_1361745791_FULL.lck
./1361000000/NetAppServer_VOBS_1361669001_FULL.lck
./1361000000/NetAppServer_1361748519_FULL.lck
./1361000000/NetAppServer_VOBS_1361755414_FULL.lck
./1361000000/NetAppServer_1361762831_FULL.lck
./1361000000/NetAppServer_VOBS_1361675098_FULL.lck
./1361000000/NetAppServer_VOBS_1361762474_FULL.lck
./1361000000/NetAppServer_VOBS_1361753519_FULL.lck
./1363000000/.lck
./1363000000/NetAppServer_VOBS_1363406404_FULL.lck
./1363000000/NetAppServer_1363406406_FULL.lck
./1363000000/NetAppServer_1363483151_FULL.lck
./1363000000/NetAppServer_1363514814_FULL.lck
./1363000000/NetAppServer_1363521660_FULL.lck
./1363000000/NetAppServer_VOBS_1363522258_FULL.lck
./1363000000/NetAppServer_VOBS_1363572380_FULL.lck
./1363000000/NetAppServer_VOBS_1363406401_FULL.lck
./1364000000/NetAppServer_VOBS_1364011203_FULL.lck
./1364000000/.lck
./1364000000/NetAppServer_VOBS_1364593630_FULL.lck
./1364000000/NetAppServer_1364630873_FULL.lck
./1364000000/NetAppServer_1364631502_FULL.lck
./1364000000/NetAppServer_VOBS_1364644490_FULL.lck
./1364000000/NetAppServer_VOBS_1364649309_FULL.lck
./1364000000/NetAppServer_VOBS_1364743799_FULL.lck
./1364000000/NetAppServer_1364694560_FULL.lck
./1364000000/NetAppServer_VOBS_1364743797_FULL.lck
./1364000000/NetAppServer_1364875205_INCR.lck
./1364000000/NetAppServer_VOBS_1364878359_FULL.lck
./1364000000/NetAppServer_VOBS_1364933862_FULL.lck
./1364000000/NetAppServer_1364939654_INCR.lck
./1364000000/NetAppServer_1364940128_INCR.lck
./1364000000/NetAppServer_1364940481_INCR.lck
./1365000000/NetAppServer_1365048004_INCR.lck
./1365000000/NetAppServer_VOBS_1365220804_FULL.lck
./1365000000/NetAppServer_VOBS_1365048375_FULL.lck
./1365000000/NetAppServer_VOBS_1365102338_FULL.lck
./1365000000/NetAppServer_1365107550_INCR.lck
./1365000000/NetAppServer_1365108872_INCR.lck
./1365000000/NetAppServer_1365109308_INCR.lck
./1365000000/NetAppServer_1365512515_INCR.lck
./1365000000/.lck
./1365000000/NetAppServer_VOBS_1365307505_FULL.lck
./1365000000/NetAppServer_1365366105_FULL.lck
./1365000000/NetAppServer_VOBS_1365607837_FULL.lck
./1365000000/NetAppServer_VOBS_1365512962_FULL.lck
./1365000000/NetAppServer_1365652803_INCR.lck
./1365000000/NetAppServer_1365377454_FULL.lck
./1365000000/NetAppServer_1365381134_FULL.lck
./1365000000/NetAppServer_VOBS_1365518317_FULL.lck
./1365000000/NetAppServer_1365576112_INCR.lck
./1365000000/NetAppServer_1365576579_INCR.lck
./1365000000/NetAppServer_1365577143_INCR.lck
./1365000000/NetAppServer_VOBS_1365607835_FULL.lck
./1365000000/NetAppServer_VOBS_1365675542_FULL.lck
./1365000000/NetAppServer_VOBS_1365675303_FULL.lck
./1365000000/NetAppServer_VOBS_1365680032_FULL.lck
./1365000000/NetAppServer_1365755584_INCR.lck
./1365000000/NetAppServer_VOBS_1365675301_FULL.lck
./1365000000/NetAppServer_1365757509_INCR.lck
./1365000000/NetAppServer_1365757782_INCR.lck
./1365000000/NetAppServer_1365755923_INCR.lck
./1365000000/NetAppServer_1365758467_INCR.lck
./1365000000/NetAppServer_1365755582_INCR.lck
./1365000000/NetAppServer_VOBS_1365825604_FULL.lck
./1365000000/NetAppServer_VOBS_1365831473_FULL.lck
./1365000000/NetAppServer_VOBS_1365899987_FULL.lck
./1365000000/NetAppServer_VOBS_1365825601_FULL.lck
./1365000000/NetAppServer_1365986276_FULL.lck
./1365000000/NetAppServer_1365986950_FULL.lck
./1365000000/NetAppServer_1365992298_FULL.lck
./1366000000/NetAppServer_VOBS_1366084804_FULL.lck
./1366000000/NetAppServer_1366084806_INCR.lck
./1366000000/NetAppServer_VOBS_1366088389_FULL.lck
./1366000000/NetAppServer_VOBS_1366156741_FULL.lck
./1366000000/NetAppServer_1366160421_INCR.lck
./1366000000/NetAppServer_1366160964_INCR.lck
./1366000000/NetAppServer_VOBS_1366084801_FULL.lck
./1366000000/NetAppServer_1366161333_INCR.lck
./1366000000/NetAppServer_1366161591_INCR.lck
./1366000000/NetAppServer_1366084803_INCR.lck
./1366000000/.lck
./1366000000/NetAppServer_VOBS_1366257605_FULL.lck
./1366000000/NetAppServer_1366257607_INCR.lck
./1366000000/NetAppServer_VOBS_1366262441_FULL.lck
./1366000000/NetAppServer_VOBS_1366268064_FULL.lck
./1366000000/NetAppServer_1366341743_INCR.lck
./1366000000/NetAppServer_VOBS_1366257601_FULL.lck
./1366000000/NetAppServer_1366342144_INCR.lck
./1366000000/NetAppServer_1366342273_INCR.lck
./1366000000/NetAppServer_1366342592_INCR.lck
./1366000000/NetAppServer_1366257603_INCR.lck
./1366000000/NetAppServer_VOBS_1366430403_FULL.lck
./1366000000/NetAppServer_1366430405_FULL.lck
./1366000000/NetAppServer_VOBS_1366518717_FULL.lck
./1366000000/NetAppServer_VOBS_1366610807_FULL.lck
./1366000000/NetAppServer_VOBS_1366430400_FULL.lck
./1366000000/NetAppServer_1366689606_INCR.lck
./1366000000/NetAppServer_VOBS_1366689976_FULL.lck
./1366000000/NetAppServer_1366862404_INCR.lck
./1366000000/NetAppServer_1366735628_INCR.lck
./1366000000/NetAppServer_1366747680_INCR.lck
./1366000000/NetAppServer_1366748095_INCR.lck
./1366000000/NetAppServer_1366750777_INCR.lck
./1366000000/NetAppServer_1366689603_INCR.lck
 

Thank you for your help,
DR

 

 

 

Marianne's picture

I cannot tell you why, just that it's normal.

Your problem is not with .lck files - the STREAMS lck file is 16 bytes and the image lck files are 0 bytes.

The image 'files' files are taking up the space. 

Please see section in the manual that I mentioned above.

Supporting Storage Foundation and VCS on Unix and Windows as well as NetBackup on Unix and Windows
Handy NBU Links

Nicolai's picture

Take a look at this thread about .lck files.

https://www-secure.symantec.com/connect/forums/netbackupdbimagesclientxxxxxx-directories-and-lck-file

They are important and needed.

According to this tech note say .lck files are :

Used to prevent simultaneous updates on images.

http://www.symantec.com/docs/HOWTO68031

Nothing indicate a problem with the Catalog - I recommend you expand the file system.

Assumption is the mother of all mess ups.

If this post answered your'e qustion -  Please mark as a soloution.

mph999's picture

As per the excellent advice from Marianne and Nicolai, your issue is caused by the size of the .f files, which are directly related to how many files are in a backup.

The best option is to extend the filesystem, the 2nd best option is to archive the catalog, the 3rd best option is to compress it.   All of these are best explained in the admin guide.

Martin

 

Regards,  Martin
 
Setting Logs in NetBackup:
http://www.symantec.com/docs/TECH75805
 
KDob's picture

The number one best way to be sure everything is normal:

1.) Be sure that you did not inadvertantly leave the NOexpire touch file in the following dir:

/usr/openv/netbackup/bin

2.) Run: /usr/openv/netbackup/bin/admincmd/nbdelete -allvolumes -force

3.) Run: /usr/openv/netbackup/bin/admincmd/bpimage -prunetir -allclients

4.) Examine whether or not the /usr/openv/netbackup/db/images folder has gone down in size

If not, then the cheapest solution is to archive off the catalog via bpcatarc.

Please see the three related commands and their proper use in the NBU 7.5 Commands PDF:

See bpcatlist on page 69.

See bpcatres on page 72.

See bpcatrm on page 73.