Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

Control number of AIR replications concurrently

Created: 12 Jun 2013 | 14 comments

I'm running AIR replications between my primary DC and a colocation.  I had replications slowing to a crawl and then piling up.  So as a test I cancelled all the replications and restarted - and only allowed *one* to run.  Throughput is quite good.  So I have a couple questions:

1) Is there any way to "throttle" the *number* of replications which concurrently run?  I would really like to *just* throttle replications.  I'd rather avoid trying to manage the number via "Max I/O streams" on the MSDP pools.

2) How many concurrent replications do you generally have during/after a backup window?  When things were really bound up I'd have ~40 active as well as a number of queued replications.

Environment: Main DC

Solaris Master

3 x 5220 Symantec Appliances

Environment: Colo

5220 master/media
Operating Systems:

Comments 14 CommentsJump to latest comment

Gautier Leblanc's picture

Hello,

I think that your only choice is to set up Max I/O streams on target MSDP Pool, sorry :-(

Mark_Solutions's picture

Not easy to acheive but you could play with a few settings - it all depends on what else you do as some setting affect everything.

If the only thing your SLPs do is replication then the LIFECYLE_PARAMETERS file could be edited so that the minimum size is very large and the minimum time to wait for small images is also very large and to group images

That way it could gather up everything into one large replication job - you could also schedule SLPs as active / inactive to get the above to work to its best.

Most settings tend to affect things globally but for AIR replication there is one setting specific to that, but only for bandwidth throttling and relates to the total bandwidth that all replications are allowed to use

This is the /disk/etc/puredisk/agent.cfg file and well down that file is bandwidthlimit = 0

The setting should be in kb/s but really needs to be played with - also needs a full re-start (or reboot) to take effect.

Realisticly the Max I/O or SLP Lifecycle Parameters is the place to look

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

elanmbx's picture

So I configured the "Max I/O Streams" on the target MSDP pool at the colocation to 15 (from unlimited).  Cancelled my replications so they would be re-initiated... and now I have 36 replications running in my admin console from my source domain.

I'm not exactly sure if this means my throttle with "Max I/O Streams" isn't working?  Is there any way to confirm?

Thanks,

-Mike-

Gautier Leblanc's picture

Have you restarted Netbackup Services on target storage server ?

elanmbx's picture

I have not.  I was under the impression that this was *not* necessary when modifying disk pool properties such as this?

Brook Humphrey's picture

This technote should answer your questions pretty good:

http://www.symantec.com/docs/TECH204574

More or less it's what is stated by others above but gives more technical details.

If you have any other questions please feel free to ask.

Thanks

Brook Humphrey
Managed Backup Service

Principal Backup Administration Specialist

elanmbx's picture

That Tech Note seems to indicate that I should be limiting the *source* disk pool's "Max I/O streams" parameter...

Will limiting the *destination* MSDP's maximum I/O setting NOT have any impact on the number of replications that run?

-Mike-

Mark_Solutions's picture

I am guessing that as you are using AIR the source has no idea of the settings in NetBackup on the destination pool

As such it just kicks in and throws data straight at the destination appliance, only when it has arrived is the Master informed of anything - it all happens Media Server to Media Server so settings on the destination are probably totally unknown

Looks like it may be lifeycle parameters then?!

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

elanmbx's picture

Unfortunately my SLPs are doing quite a lot of operations.  So changing LIFECYCLE_PARAMETERS will definitely impact more than just replications.

It would be kind of nice if there were some LIFECYCLE_PARAMETERS config options to control Replications independently of other SLP operations...

Brook Humphrey's picture

Yeah as teh technote states you can limit max streams but if you want more streams you need to limit the size of the SLP duplicaion by modifying the LIFECYCLE_PARAMETERS and yes this is on the source side not the destination.

Thanks

Brook Humphrey
Managed Backup Service

Principal Backup Administration Specialist

Mark_Solutions's picture

Looks like it is actually even worse!! Reading through this tech note http://www.symantec.com/docs/TECH204574 I see that each replication uses 4 streams - this can be adjusted in the agent.cfg

I wonder what else can be configured in there??

This tech note does confirm the I/O is on the source de-dupe pool for limiting I/O

Authorised Symantec Consultant

Don't forget to "Mark as Solution" if someones advice has solved your issue - and please bring back the Thumbs Up!!.

elanmbx's picture

What is interesting is that I limited the *destination* MSDP to "Max I/O streams = 8" and it *seems* to be providing a limit on the source side... I only have 18 concurrent replications active and *NONE* queued.

Not sure what exaclty to make of it... but I'm going to continue to monitor progress and see if I'm just jumping the gun with regards to my conclusion.

elanmbx's picture

Looks like I jumped the gun.  Some of the replications apparently took awhile to get started.  Right now I have 36 replications in active state.  20 are coming from a single source MSDP (I have 3 total MSDPs on the source side).

They all seem to be running OK... so I will simply continue to monitor.

elanmbx's picture

Just trying to manually control the number of concurrent AIR replications... I've killed all but 5 per appliance and throughput appears to be getting *way* better.

I sure wish there was a way to control the number of replications that get started per appliance.