The images stay on disk as long as possible (or until they expire)

The disk images are copy 1 and are primary

The tape images are copy 2 and only become primary once the disk images are expired

Are you actually seeing failures as a result of the disk filling up?

http://www.symantec.com/docs/TECH66149

This doc covers disk cleanup behavoir in NetBackup - the key being the difference between the High and Low watermarks

However, if you are not duplicating your backups efficiently enough then NetBackup cannot do the cleanup and so will grow above the High Watermark, eventually filling the disk when backups will fail

The setting are just triggers and reply on verything else being in order to be of any effect

Hope this helps

Did this every work ?

If it did, what has changed ?

Martin

Also realised you use bpimage -cleanup -allclients, but if they are not expired they will not cleanup from disk

If you want to cleanup expired disk images only use: nbdelete -allvolumes -force

But again, if they have not expired they won't go

Staging only clears down images when it has hit the high watermark and only then on the condition that the images have been duplicated to tape os make sure your duplications are all up to date and refer to the tech note in relation to High and Low Watermarks - what settings are you using and what size is the staging area?

The disk is full and already crossed the high water mark

Since this is a temporary staging area I do not see a reason for the images to remain on disk after relocation has successfully completed

No settings were changed as this has worked for last 2 yrs

Currently I am manually marking all relocated images to tape as primary and expiring the images in dssu

This is by design - the disk staging area will remain full at all times (at least up to the High Water Mark)

This allows any restores needed to be done quickly from disk rather than from tape

If you expire everything it wil just fill up again

If you are running manual relocations and duplications are kicking in then your duplications are not up to date and you should look at your staging schedule

If you are not actually getting any Status 129 - Disk Storage Unit Is Full errors then there is nothing wrong with your setup

Is this a cluster, or, aare there multiple interfaces on the master ?

OK, if you look in the expmgr log ( 272) (Debug / Diag at 6, do you see a High Water Mark event.

Something like this :

16/08/2011 22:29:40.160 V-272-6 [ExpMgrEventConsumer::notify] High Water Mark event received

vxlogview should be run with the options -d all -i 272

The logs involved for this part :

272 - expmgr (this is written into the 226 log)

226 - nbstserv 

111 - nbemm

230 - rdsm

178 - dsm

mkdir /usr/openv/netbackup/logs/admin

Add VERBOSE = 5 to bp.conf

Without testing this, there couold be other logs involved, I think to start with .

1 Enable the 272 / 226/ 111  logs at debug/ diag 6

2 Create the 'fault' that is, allow the disk space to fill up to the HWM

3  Grab logs, run through vxlogview as shown 

-i 111 -d all for the emm log

-i 226 -d -all for nbstserv log

+ admin log.

This is from memory, not able to test/ check at the moment, so apologies if I have made a mistake ...

Thanks,

Martin

I would like to see a bpdm log as well...

NBU 6.5 DSSU cleanup behaviour is explained in this TN: http://www.symantec.com/docs/TECH66149

What kind of storage are you using for DSU?

Is it a dedicated lun / mount point for DSU only and not a repository for other kinds of data as well?

Hi Marianne

The understanding about highwater mark and low water mark explained in the shared article is correct.

We have tried lowering the HWM to 10 from a previous value of 75 but the images refused to cleanup automatically.

Please find the bpbrmds, bpdm and vxlogmgr logs in the below mentioned fileshare link

https://fileshare.symantec.com

Folder - SymantecConnect

userid - Sagar_K
passwd - Sym@nt3c

Please Note-Logs have been compressed using 7z application which can be freely downloaded from http://www.7-zip.org/

Regards

Sammeer

Hi Sammeer,

Due to a bit of confusion, we actually meant to ask also for the bpdm log - please accept out apologies for this.

We need this log, for the same time period as the other logs.

Martin

OK, I see you sent the bpdm log already - thx.

The HWM event is seen in the 226 log (74971 times in fact ... ) so I think we can say it works ...

I don't know which  STU is not clearing down ???   but ...

Looking in bpdm ...

HighWater Mark 10, Low Water Mark 5

This could strangle the system a little to be honest - I understand what you are trying but these settings are extreme and NetBackup may just not be able to do anything with them

Can I just clarify two things please...

1. Are all of your duplications kept up to date - what is the window on your disk staging schedule?

2. Could you give us the output of:

nbdevquery -listdv -D -stype BasicDisk

Just to double check everything

Hi Mark

1. Are all of your duplications kept up to date - what is the window on your disk staging schedule?
- We have a daily schedule for relocation set to 10am

Please find the below output of nbdevquery -listdv -D -stype BasicDisk

Disk Volume Dump
  name                : <Internal_16>
  id                  : <G:\netback9_Monthly_dssu>
  diskpool            : <netback9_Monthly_dssu::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaam>
  total_capacity      : 2013384003584
  total_phys_capacity : 2013384003584
  free_space          : 1056105869312
  free_phys_space     : 957278134272
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xf
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </dss2>
  diskpool            : <Dakcmstnetbkp_dssu2::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaaI>
  total_capacity      : 993989670912
  total_phys_capacity : 993989670912
  free_space          : 856050805760
  free_phys_space     : 137938865152
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : <S:\>
  diskpool            : <netback9_full_dssu::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaa3>
  total_capacity      : 982978129920
  total_phys_capacity : 982978129920
  free_space          : 941469016064
  free_phys_space     : 41509113856
  potential_free_space: 51280728064
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xf
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </dss4>
  diskpool            : <Dakcmstnetbkp_dssu3::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaaj>
  total_capacity      : 1932156461056
  total_phys_capacity : 1932156461056
  free_space          : 1923511672832
  free_phys_space     : 8644788224
  potential_free_space: 14720761856
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : <G:\netback9_Daily_dssu>
  diskpool            : <netback9_Daily_dssu::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaan>
  total_capacity      : 2013384003584
  total_phys_capacity : 2013384003584
  free_space          : 1056105869312
  free_phys_space     : 957278134272
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xf
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </catalog/HSL_Daily>
  diskpool            : <Dakcmstnetbkp_dssu_HSL_Daily::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaat>
  total_capacity      : 105432537088
  total_phys_capacity : 105432537088
  free_space          : 105318130688
  free_phys_space     : 114406400
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </catalog/HSL_Weekly>
  diskpool            : <Dakcmstnetbkp_dssu_HSL_Weekly::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaas>
  total_capacity      : 105432537088
  total_phys_capacity : 105432537088
  free_space          : 105318130688
  free_phys_space     : 114406400
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : <G:\netback4_full_dssu>
  diskpool            : <netback4_full_dssu::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaaz>
  total_capacity      : 982970277888
  total_phys_capacity : 982970277888
  free_space          : 940054462464
  free_phys_space     : 42915815424
  potential_free_space: 18360924160
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : <S:\Weekly_Dssu\>
  diskpool            : <netback9_Weekly_dssu::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaa6>
  total_capacity      : 982978129920
  total_phys_capacity : 982978129920
  free_space          : 941469016064
  free_phys_space     : 41509113856
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xf
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : <S:\Monthly_Dssu\>
  diskpool            : <netback9_monthly_dssu::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaa7>
  total_capacity      : 982978129920
  total_phys_capacity : 982978129920
  free_space          : 941469016064
  free_phys_space     : 41509113856
  potential_free_space: 4748288
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xf
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </dss2/Weekly_Dssu/>
  diskpool            : <Dakcmstnetbkp_dssu2_Weekly::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaa8>
  total_capacity      : 993989670912
  total_phys_capacity : 993989670912
  free_space          : 856050805760
  free_phys_space     : 137938865152
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </dss2/Monthly_Dssu/>
  diskpool            : <Dakcmstnetbkp_dssu2_Monthly::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaa9>
  total_capacity      : 993989670912
  total_phys_capacity : 993989670912
  free_space          : 856050805760
  free_phys_space     : 137938865152
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </dss4/Weekly_Dssu/>
  diskpool            : <Dakcmstnetbkp_dssu3_Weekly::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaaC>
  total_capacity      : 1932156461056
  total_phys_capacity : 1932156461056
  free_space          : 1923511672832
  free_phys_space     : 8644788224
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </dss4/Monthly_Dssu/>
  diskpool            : <Dakcmstnetbkp_dssu3_Monthly_::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaaD>
  total_capacity      : 1932156461056
  total_phys_capacity : 1932156461056
  free_space          : 1923511672832
  free_phys_space     : 8644788224
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : </dss4/impfiles/>
  diskpool            : <Dakcmstnetbkp_dssu3_impfiles::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaaE>
  total_capacity      : 1932156461056
  total_phys_capacity : 1932156461056
  free_space          : 1923511672832
  free_phys_space     : 8644788224
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : <G:\netback4_weekly\>
  diskpool            : <netback4_weekly_dssu::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaaF>
  total_capacity      : 982970277888
  total_phys_capacity : 982970277888
  free_space          : 940054462464
  free_phys_space     : 42915815424
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>
Disk Volume Dump
  name                : <Internal_16>
  id                  : <G:\netback4_monthly\>
  diskpool            : <netback4_monthly_dssu::dakcmstnetbkp::BasicDisk>
  disk_media_id       : <@aaaaG>
  total_capacity      : 982970277888
  total_phys_capacity : 982970277888
  free_space          : 940054462464
  free_phys_space     : 42915815424
  potential_free_space: 0
  committed_space     : 0
  precommitted_space  : 0
  nbu_state           : 1
  sts_state           : 0
  flags               : 0xe
  num_read_mounts     : 0
  max_read_mounts     : 0
  num_write_mounts    : 1
  max_write_mounts    : 1
  system_tag          : <Generic disk volume>

Could you please cover all of the following points:

1. Could you clarify you staging window please - you say 10am each morning - what is the actual window and start time? Could you please provide the output of:

bpschedule -U

2. Which of the Disk Storage units do you have issues with?

I see that some have a figure for potential free space and some do not.

3. Please also provide the output of bpstulist -U

4. Do you actually experience failures?

Thanks

Hi Mark

1. Could you clarify you staging window please - you say 10am each morning - what is the actual window and start time? Could you please provide the output of:

bpschedule -U

- Output shared on Fileshare Userid - Sagar_K - Password - Sym@nt3c Folder name - SymantecConnect

2. Which of the Disk Storage units do you have issues with?

I see that some have a figure for potential free space and some do not.

- We are experiencing DSSU issue on all Disk Storage Units - even the ones that are connected to lan media servers.

3. Please also provide the output of bpstulist -U

- Output shared on Fileshare Userid - Sagar_K - Password - Sym@nt3c Folder name - SymantecConnect 

4. Do you actually experience failures?

We do not experience any failures since we are performing manual cleanup of all images.

WOW! - so lets do these in importance order!!

Firstly these:

Dakcmstnetbkp_dssu2_Monthly
no window

Dakcmstnetbkp_dssu2_Weekly
no window

Dakcmstnetbkp_dssu3_Monthly_
no window

Dakcmstnetbkp_dssu3_Weekly
no window

netback9_Daily_dssu
no window

netback9_Monthly_dssu
no window

So none of these automatically duplicate any images - unless you run a manual relocation they do not go to tape so cannot clear images down automatically. If you do not manually relocate them and have exoired images to clear disk space then you have lost those backups.

Next this one:

Dakcmstnetbkp_dssu2
full window every 12 hours

This will kick in twice a day - so not too bad but see my reccomendations at the end

Next these:

Dakcmstnetbkp_dssu3
9am to 3pm every 10 hours

netback4_full_dssu
7am to 1pm every 12 hours

netback4_monthly_dssu
7am to 11am every 12 hours

netback4_weekly_dssu
7am to 11am every 12 hours

netback9_Weekly_dssu
7am to 11am every 12 hours

netback9_full_dssu
7am to 1pm every 12 hours

netback9_monthly_dssu
7am to 11am every 12 hours

Not great but as long as backups have all finished by 7am / 9am at least they run a duplication batch daily

Next these:

Dakcmstnetbkp_dssu_HSL_Weekly
8am to 10am every 7 days

Dakcmstnetbkp_dssu_HSL_Daily
8am to 10am every 7 days

These only run once a week but as you have every day specified they could run on any day so may not be able to clear anything down for a week.

So reccomendations ....

Set a window to the schedules for all those that do not have one

If you want the system to be almost as efficient as a SLP then set all DSSUs to have a full window (or at least a window from about an hour after your backups start in the evening to mid morning) with a frequency of 2 or 3 hours. If they do not have anything to do they will just run the parent job and finish with a blue man - far better than being behind on the duplications

I would reccomend changing ALL of you staging schedules immediately - I have the feeling based on what you have said that there is a possibility that you have lost data here.

Hope this helps