Video Screencast Help

Out of FS space for /usr/openv (NBU Production Server)

Created: 18 May 2009 • Updated: 21 May 2010 | 7 comments
This issue has been solved. See solution.

I received an error from my Device Monitor - database system error. DSM is unable to connect to the DB, issue: connect  - I have ran out of disk space and need to have our storage group "grow" more space for my FS. Netbackup 6.5.3 daemons are currently at a "stop" state (inc. Media Servers). I was told that there is a temp fix from my UNIX Admin. I am currently "mirroring" /usr/openv. Is it possible (i'm sure, not recommended) to remove this mirror for a quick fix and continue my backups? Will I need to "re-zone" all my drives? Re-configure the Robot?

Any insight would help.

Thank you

Comments 7 CommentsJump to latest comment

CY's picture

Hi Jay,

Is this one of your Media servers, or is it your NBU Master server?

If it's the Master server, I'd rather to keep the volume mirroring - whether the Unis admin configured it with H/W RAID (preferred way), or using S/W mirroring such as VxVM.  It's too bad you run out of disk space. Try reducing NBU logging level and log retention days to save disk space until your Unix admin can get more disk space for you.

If it's just one of the Media servers, I'd say you can break the mirror for now and extend your volume and F/S.  It certainly a good practice to have the /usr/openv mirrored, but if you can't get additional disk space for now, it is better to have some availability than zero availability.

The daemons are at "stop" state because they can't write logs any more.  Once you get enough space on the file system, stop and restart NetBackup on the Media server should bring them back to operation.  I don't think you need to reconfigure drives or robot/library - they are logically configured in NetBackup EMM DB, so unless your EMM DB (most likely on your Master server) is corrupted there is no reason to reconfigure them.

Even the worst case, you don't need to re-zone them (unless you replaced HBA on the media server, AND the new HBA can't re-claim the old WWPN from the old HBA. Actually, most of the new HBA can use software to change their WWPN.)

Jay Son's picture

Its my Master Server (aaaargh!)

Thanks for your quick response and vital information. Logging Level is already at 0. I will work with my Storage group and allocate more space. I too was hesitant on breaking the "mirror".

Thanks again
Jay

quebek's picture

Hi
Breaking mirrors is not recommended but can be done ;)
Once You will break Your mirror, ask UNIX guys to do it, and later on tell them to assgin freed disk space to /usr/openv.
No You will not have to re-zone Your drives, nor reconfigure robot, You will just incerese space on /usr/openv but removing mirror and adding this disks to the volume group to extend Your filesystem
Also You can try to remove some logs from /usr/openv/netbackup/logs, or to move Your catalog to different FS. Follow the chapter "Moving the image catalog" on page 354 from Admin guide for Unix Vol 1

CY's picture

Run du -sk on /usr/openv/*, as well as on its subdirectories, to see which directory takes the most of the space.

It is recommended to put the uniified logs (/usr/openv/logs), as well as the NBU catalogs (/usr/openv/netbackup/db) on their own file systems.

I also have the EMM DB (/usr/openv/db/data), as well as the legacy logs (/usr/openv/netbackup/logs) on their own file systems, respectively.

Now it's time for you to talk to the Unix admin to get enough disk space allocated for the above directories/file systems.  :-)

bak's picture

Another option to save space is to implement compression on your catalog.  This just means that when you go to restore it first has to uncompress the meta data for that backup.  I would definitely want to keep the catalog data mirrored.  Ideally the catalog resides on a storage array where additional space can be allocated dynamically.

Command to enable compression for anything older than a week:
/usr/openv/netbackup/bin/admincmd/bpconfig -cd 604800

An argument of 0 will disable this feature.  You can view current settings by running this command:
/usr/openv/netbackup/bin/admincmd/bpconfig -L

Stuart Green's picture

2 quick space savers on a master /usr/openv filesystem

  1. Unified logging files
  2. Old Update Packs

1. Unified logs. Turn logging down to a minimal and purge them.
Read this forum post for all the best ways to reclaim space.
https://www-secure.symantec.com/connect/forums/usr...

2. /usr/openv/pack/
In here you might find saved backups of previous updates or maintenance packs
Generally, I delete everything in here except any references to Prior Update Versions
You can reclaim a big chunk of space. If you updated to 6.5.1 then 6.5.2 then 6.5.3 etc you can clean out those folders by now as not needed.
Or back them up prior to deleting.
Folders with names such as
./NB_CLT_6.5.3
./NB_6.5.3
./NB_JAV_6.5.3
These contain backups of binaries prior to updating to 6.5.3
So if you were on 6.5.4 you could delete them, but leave the
./NB_CLT_6.5.4
./NB_6.5.4
./NB_JAV_6.5.4
directories in case you ever uninstall 6.5.4 and revert to previous 6.5.3 for example

Or if /usr/openv/ on a volume in a volume/disk group ask them to add a new disk into the volume group and extend the file system if possible,

Tip: Get overview/document your NBU environment. Run 'nbsu' and review the output.

• If this provides help, please vote or mark appropriate solution.

SOLUTION
DavidParker's picture

Nice info, Stuart!

I've been continually annoyed by the disk space 'pack' files keep taking up, as well as the apparent lack of control on the growth of the software in general (the unix client patch is what, 900MB now? sheesh!).

In my environment, the /usr directory is strictly sized and managed.  The admins are quite annoyed when I have to ask them for an additional 500MB on X systems every 6 months.

$0.00.5