Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

How do I verify my data size that will be replicated through VVR

Created: 10 Dec 2013 • Updated: 31 Dec 2013 | 17 comments
This issue has been solved. See solution.

I need to find a way to figure out the Mb/sec (Mb=Megabits) that will be replicated to my offsite and the pipe size I need in order to stay insync. I have been testing VVR and have all of my setting in place with everything working correctly. But with a 10Mb VPN internet pipe I have the line pegged at 10Mb all the time and am never able to catchup. When I looked at Performance monitor, Physical and Logical disks I am only generating about 4Mb/sec of disk writes which is what is being replicated the the offsite. Am I looking at the wrong data for my replication size needed? Is there a way to verify the Mb/sec that are going to be written the my offsite prior to starting VVR?

NOTE: If you are having similar problems make sure to read the whole thread, there were mutliple solutions that were needed to verify and fix the problem of the data never catching up completely.

 

Operating Systems:

Comments 17 CommentsJump to latest comment

Wally_Heim's picture

Hi mhab11,

We do have a utility called vradvisor which can be used to properly size your configuration.  It is located in the \Tools\storage_foundation folder on the media in 6.0.1 release (I think it is in the same location on most release media.)

It is best used with an I/O data collection of at least 2 weeks.  This would give the best results of what I/O your configuration generates on the volumes being replicated.  The longer the collection period the better the results.

When you do your initial sync of your replicated volumes VVR has to copy all the data that is currently on the volume(s) to the remote site.  This process can overwhelm replication links and can take many days or weeks to complete..

You can use the following command to see how much data is needing to be replicated in 5 second intervals:

   vxrlink -i 5 status <rlink_name>

The rlink name is available by running this command:

   vxrlink -VPl

The rate of replication and errors can be seen with this command.

   vxrlink -e -i 5 stats <rlink_name>

I often pipe the stats command to a text file and look at it after its been running for 10+ minutes.

Also the first line of the stats output should be ignored.  It is miscalculated and does not contain good data.

if you need to check memory used by VVR you should run this command:

   vxmemstat -i 5

I hope this helps.  Please share more configuration details and the outputs of these command if you are still having problems.

Also open a case with Symantec Techncial Support if you need more indepth assistance with tuning your replication configuration or if you need alternate initial sync options.

Thank you,

Wally

 

mikebounds's picture

To see what you are writing, use vxstat:

If you think you are writing 4Mb/sec then you should see:

On the primary site, 4Mb/sec going to SRL volume and an additional 4Mb/sec for the total of all the data volumes  and on the secondary site the SRL is only used to keep track of writes so there should be a small amount to SRL and 4Mb/sec to data volumes.

Maybe you have your units mixed up and your network is correct at 10Mbits/sec, but your disk is writing 4Mbytes/sec.  

vxstat reports in 512k block so multiply by 2048 to get Mbytes or multiply by 16384 to get Mbits.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

SOLUTION
mhab11's picture

Thanks, I did a vxsts -g diskgroup -i 2 to get a continues print so I could see what my range was. my Blocks Write ranges from 650 to 1000 for the volume I am replicating. So that is 10649600Mbps?? I did mean Mb not MB, MBx8=Mb. I do have 2 disk arrays with one VD presented from each in the volume. Would that be 2 writes is has to do therefor doubling the Blocks Write?

  Primary Server

       Operations       Blocks           Avg.Time
       Read  Write     Read  Write     Read    Write

(D:)     5       78       2269    906        15      222

Secondary server

       Operations       Blocks           Avg.Time
       Read  Write     Read  Write     Read    Write

         0        5           0      2421         0       16

 

So you can see the secondary is more then Double the Primary, this is what I cannot figure out where the excess data is coming from.

mikebounds's picture

Sorry should have been divide by 2048 for MBytes and then this need to be multiplied by 8 for Mbits (or divide original figure by 256 instead) - so

For 900 blocks for 2 seconds this is 450 blocks a second which is 450/256 = 1.76 Mbits/s

 

I don't understand your output as you should see entries for each volume - so you should see a line for srl volume as well as D:

Can you provide full output of vxstat for at least 2 iterations on both systems and also output from vxprint

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

mhab11's picture

Primary Server

Wed Dec 11 12:29:18 2013

Volume  AvMail_Snap133  0       0       0       0       0       0
Volume  AvMail_Snap134  0       0       0       0       0       0
Volume  AvMail_Snap132  0       0       0       0       0       0
Volume  AvMail (D:)     5       63      2544    714     11      182
Volume  Volume          0       63      0       783     0       97

Wed Dec 11 12:29:20 2013

Volume  AvMail_Snap133  0       0       0       0       0       0
Volume  AvMail_Snap134  0       0       0       0       0       0
Volume  AvMail_Snap132  0       0       0       0       0       0
Volume  AvMail (D:)     5       63      2544    714     11      182
Volume  Volume          0       63      0       783     0       97

Wed Dec 11 12:29:22 2013

Volume  AvMail_Snap133  0       0       0       0       0       0
Volume  AvMail_Snap134  0       0       0       0       0       0
Volume  AvMail_Snap132  0       0       0       0       0       0
Volume  AvMail (D:)     4       79      2064    934     7       166
Volume  Volume          0       80      0       1025    0       76

Wed Dec 11 12:29:24 2013

Volume  AvMail_Snap133  0       0       0       0       0       0
Volume  AvMail_Snap134  0       0       0       0       0       0
Volume  AvMail_Snap132  0       0       0       0       0       0
Volume  AvMail (D:)     4       79      2064    934     7       166
Volume  Volume          0       80      0       1025    0       76

 

Secondary Server

Volume  REP             0       0       0       0       0       0
Volume  AvMail          0       4       0       1970    0       16

Wed Dec 11 12:34:08 2013

Volume  REP             0       0       0       0       0       0
Volume  AvMail          0       4       0       1970    0       16

Wed Dec 11 12:34:10 2013

Volume  REP             0       0       0       0       0       0
Volume  AvMail          0       5       0       2421    0       16

mhab11's picture

Primary

Diskgroup = Archive
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg Archive      Archive      -        -        -        -        -       -
dm Harddisk11   LanAcesLogs2 -        838849977-        -        -       -
dm Harddisk6    LanAcesLogs1 -        838849977-        -        -       -
sd LanAcesLogs1-01 MailArchive-01   ENABLED  734003200 0        -        -
 -
sd LanAcesLogs2-01 MailArchive-02   ENABLED  734003200 0        -        -
 -
v  MailArchive  -            ENABLED  734003200-        ACTIVE   -       -
pl MailArchive-01 MailArchive  ENABLED  734003200 -        ACTIVE   -       -
pl MailArchive-02 MailArchive  ENABLED  734003200 -        ACTIVE   -       -

Diskgroup = AvMailDiskGroup
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
sd 0-1-01       AvMailVolume-03   ENABLED  1258291200 0        -        -
-
sd 0-1-02       AvMailVolume_DCO1-02   ENABLED  4096     0        -        -
   -
sd 0-1-03       AvMailVolume-06   ENABLED  512      0        -        -       -
sd 0-2-01       Snap1-01     ENABLED  1258291200 0        -        -       -
sd 16-1-01      AvMailVolume-05   ENABLED  1258291200 0        -        -
-
sd 16-1-02      AvMailVolume_DCO1-04   ENABLED  4096     0        -        -
   -
sd 16-2-01      Snap3-01     ENABLED  1258291200 0        -        -       -
sd 8-1-01       AvMailVolume-04   ENABLED  1258291200 0        -        -
-
sd 8-1-02       AvMailVolume_DCO1-03   ENABLED  4096     0        -        -
   -
sd 8-1-03       AvMailVolume-07   ENABLED  512      0        -        -       -
sd 8-2-01       Snap2-01     ENABLED  1258291200 0        -        -       -
dg AvMailDiskGroup AvMailDiskGroup     -        -        -        -        -
   -
v  AvMailVolume -            ENABLED  1258291200-        ACTIVE   -       -
pl AvMailVolume-01 AvMailVolume ENABLED  1258291200 -        ACTIVE   -       -
pl AvMailVolume-02 AvMailVolume ENABLED  1258291200 -        ACTIVE   -       -
pl AvMailVolume-03 Snap1        ENABLED  1258291200 -        ACTIVE   -       -
pl AvMailVolume-04 Snap2        ENABLED  1258291200 -        ACTIVE   -       -
pl AvMailVolume-05 Snap3        ENABLED  1258291200 -        ACTIVE   -       -
pl AvMailVolume-06 AvMailVolume ENABLED  512      -        ACTIVE   -       -
pl AvMailVolume-07 AvMailVolume ENABLED  512      -        ACTIVE   -       -
sd Disk1-01     AvMailVolume-02   ENABLED  1258291200 0        -        -
-
dm Harddisk10   16-2         -        1260379512-        -        -       -
dm Harddisk2    Mirror1      -        1363146689-        -        -       -
dm Harddisk3    0-1          -        1260379512-        -        -       -
dm Harddisk4    8-1          -        1260379512-        -        -       -
dm Harddisk5    16-1         -        1260379512-        -        -       -
dm Harddisk7    Mirror2      -        1363146689-        -        -       -
dm Harddisk8    0-2          -        1260379512-        -        -       -
dm Harddisk9    8-2          -        1260379512-        -        -       -
sd Mirror1-01   AvMailVolume_DCO1-01   ENABLED  4096     0        -        -
   -
sd Mirror1-02   REP-02       ENABLED  204800   0        -        -       -
sd Mirror2-01   REP-01       ENABLED  204800   0        -        -       -
sd Mirror2-02   AvMailVolume-01   ENABLED  1258291200 0        -        -
-
rv NYVC         -            -        -        -        ACTIVE   -       -
rl ANP2NYC      NYVC         -        -        -        ACTIVE   -       -
v  REP          -            ENABLED  204800   -        ACTIVE   -       -
pl REP-01       REP          ENABLED  204800   -        ACTIVE   -       -
pl REP-02       REP          ENABLED  204800   -        ACTIVE   -       -
v  Snap1        -            ENABLED  1258291200-        ACTIVE   -       -
pl Snap1-01     Snap1        ENABLED  1258291200 -        ACTIVE   -       -
v  Snap2        -            ENABLED  1258291200-        ACTIVE   -       -
pl Snap2-01     Snap2        ENABLED  1258291200 -        ACTIVE   -       -
v  Snap3        -            ENABLED  1258291200-        ACTIVE   -       -
pl Snap3-01     Snap3        ENABLED  1258291200 -        ACTIVE   -       -

Diskgroup = WIN-MSBKUUROFN9-Dg0
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  HarddiskVolume1 -            DISABLED 204800   -        Stopped
-       -
v  HarddiskVolume2 -            ENABLED  976564224-        ACTIVE   -       -
 

 

Secondary

Diskgroup = BasicGroup
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0

Diskgroup = AvMailDiskGroup
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg AvMailDiskGroup AvMailDiskGroup     -        -        -        -        -
   -
v  AvMailVolume -            ENABLED  1258291200-        ACTIVE   -       -
pl AvMailVolume-01 AvMailVolume ENABLED  1258291200 -        ACTIVE   -       -
pl AvMailVolume-02 AvMailVolume ENABLED  512      -        ACTIVE   -       -
pl AvMailVolume-03 AvMailVolume ENABLED  512      -        ACTIVE   -       -
sd Disk1-01     AvMailVolume-01   ENABLED  629145600 0        -        -       -

sd Disk1-02     REP-01       ENABLED  204800   0        -        -       -
sd Disk1-03     AvMailVolume-02   ENABLED  512      0        -        -       -
sd Disk2-01     AvMailVolume-01   ENABLED  629145600 0        -        -       -

sd Disk2-02     AvMailVolume-03   ENABLED  512      0        -        -       -
dm Harddisk2    Disk1        -        976768002-        -        -       -
dm Harddisk3    Disk2        -        976768002-        -        -       -
rv NYVC         -            -        -        -        ACTIVE   -       -
rl NYC2ANP      NYVC         -        -        -        ACTIVE   -       -
v  REP          -            ENABLED  204800   -        ACTIVE   -       -
pl REP-01       REP          ENABLED  204800   -        ACTIVE   -       -

Diskgroup = MHSWS001NYC-3-Dg0
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  HarddiskVolume1 -            DISABLED 204800   -        Stopped
-       -
v  HarddiskVolume2 -            ENABLED  976562176-        ACTIVE   -       -

C:\Windows\system32>

mikebounds's picture

If VVR is catching up, then what is why you will see more writes at secondary - so VVR should catch up.  As Wally says you can use:

 

   vxrlink -i 5 status <rlink_name>

The rlink name is available by running this command:

   vxrlink -VPl

to verify that you are catching up so this should show you are catching up by about 1000 blocks a second  (i.e if you write 900 block at primary and 1900 - 2400 at secondary, then VVR should be catching up by the difference) - can you provide output of  vxrlink -i 2 status

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

mhab11's picture

Primary

C:\Users\mhabicht>vxrlink -i 2 status ANP2NYC
12/11/2013 14:19:06
RLINK is in autosync. Less than 1% (923552 Kbytes) remaining.
RLINK is in autosync. Less than 1% (921120 Kbytes) remaining.
RLINK is in autosync. Less than 1% (918688 Kbytes) remaining.
RLINK is in autosync. Less than 1% (918080 Kbytes) remaining.
RLINK is in autosync. Less than 1% (918688 Kbytes) remaining.
RLINK is in autosync. Less than 1% (916864 Kbytes) remaining.
RLINK is in autosync. Less than 1% (913824 Kbytes) remaining.
RLINK is in autosync. Less than 1% (912608 Kbytes) remaining.
RLINK is in autosync. Less than 1% (913216 Kbytes) remaining.
RLINK is in autosync. Less than 1% (912000 Kbytes) remaining.
RLINK is in autosync. Less than 1% (910784 Kbytes) remaining.
RLINK is in autosync. Less than 1% (910176 Kbytes) remaining.
RLINK is in autosync. Less than 1% (908352 Kbytes) remaining.
RLINK is in autosync. Less than 1% (905312 Kbytes) remaining.
RLINK is in autosync. Less than 1% (904704 Kbytes) remaining.
 

 

mikebounds's picture

This shows VVR is catching up, but it is catching from DCM, not SRL and if you only have 10MB left in DCM, then it SHOULD be clear in a matter of minutes, however, I suspect you maybe seeing an issue which I saw in UNIX VVR 5.0 which was fixed by 5.1 in for VVR on UNIX so I am surprised it is not fixed in Windows.

This is what I think the issue is:

In earlier versions of VVR on UNIX (5.0, maybe even 4.1), the DCM autosync would not catch up if there was a modest amount of writes going on as the process was to replicate the all the dirty blocks in DCM and then check DCM again and replicate any new dirty blocks so this is flawed as unless you have a quiet period where not many block are written, VVR never catches up.  This was fixed in later version by essentially putting a checkpoint in SRL so that VVR could drain writes from DCM after one (or maybe 2) passes of the DCM.

If this is the case, then if you can briefly pause the apps so it doesn't write then VVR should catch up and then it should stay up to date as your bandwidth seems high enough for how much you are writing.

If this is the issue, then Wally might know whether the same change to use the SRL at the end of DCM autosync was implemented in Windows by 5.1SP2 and if not, if this change was implemented in later 5.1 SPs or in 6.x.

Mike

 

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

mhab11's picture

And that all sounds like it. I started at about 100GB, after about 2 days it will catch up on all data except what you are seeing. I will try to pause the Application and see it is catches up as this service has traffic 24-7. Current output you can see it went back up

 

RLINK is in autosync. Less than 1% (1022656 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1020224 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1020832 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1022656 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1022048 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1021440 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1020832 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1018400 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1019616 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1016576 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1016576 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1014752 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1012928 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1011712 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1011104 Kbytes) remaining.
RLINK is in autosync. Less than 1% (1008064 Kbytes) remaining.

mikebounds's picture

In answer to your original question - you can run vxstat over an extended period to see what bandwidth requirements are, either by calculating manually or by feeding vxstat output file into vradvisor which is available by download for any version of VVR.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

mhab11's picture

I stopped the application for 3 min and it did start to replicate faster.

NK is in autosync. Less than 1% (728992 Kbytes) remaining.
NK is in autosync. Less than 1% (726560 Kbytes) remaining.
NK is in autosync. Less than 1% (722912 Kbytes) remaining.
NK is in autosync. Less than 1% (721696 Kbytes) remaining.
NK is in autosync. Less than 1% (718048 Kbytes) remaining.
NK is in autosync. Less than 1% (715616 Kbytes) remaining.
NK is in autosync. Less than 1% (713184 Kbytes) remaining.
NK is in autosync. Less than 1% (711968 Kbytes) remaining.
NK is in autosync. Less than 1% (710144 Kbytes) remaining.
NK is in autosync. Less than 1% (707712 Kbytes) remaining.
NK is in autosync. Less than 1% (704672 Kbytes) remaining.
NK is in autosync. Less than 1% (703456 Kbytes) remaining.
NK is in autosync. Less than 1% (699808 Kbytes) remaining.
NK is in autosync. Less than 1% (699200 Kbytes) remaining.

But I still have 1.4million blocks to replicate. in the 3 min I only replicated 400k so just over 100k per min.

Application restarted the pending blocks jumped right back up

RLINK is in autosync. Less than 1% (680352 Kbytes) remaining.
RLINK is in autosync. Less than 1% (745408 Kbytes) remaining.
RLINK is in autosync. Less than 1% (782496 Kbytes) remaining.
RLINK is in autosync. Less than 1% (814112 Kbytes) remaining.
RLINK is in autosync. Less than 1% (860320 Kbytes) remaining.
RLINK is in autosync. Less than 1% (874912 Kbytes) remaining.
RLINK is in autosync. Less than 1% (893760 Kbytes) remaining.
RLINK is in autosync. Less than 1% (909568 Kbytes) remaining.
RLINK is in autosync. Less than 1% (919904 Kbytes) remaining.
RLINK is in autosync. Less than 1% (927808 Kbytes) remaining.
RLINK is in autosync. Less than 1% (936320 Kbytes) remaining.
RLINK is in autosync. Less than 1% (939968 Kbytes) remaining.
RLINK is in autosync. Less than 1% (942400 Kbytes) remaining.
RLINK is in autosync. Less than 1% (944224 Kbytes) remaining.
RLINK is in autosync. Less than 1% (947872 Kbytes) remaining.
RLINK is in autosync. Less than 1% (950912 Kbytes) remaining.
RLINK is in autosync. Less than 1% (953952 Kbytes) remaining.
RLINK is in autosync. Less than 1% (957600 Kbytes) remaining.

 

Thanks for the help I see where my problem is now, which is why I was confused looking at my data size. I will open a ticket with Symantec to see if there is a patch in a newer CP, I am running CP16.

mikebounds's picture

ok - let me know outcome of ticket with Symantec and if Symantec can't find details of issue that was fixed in UNIX, then let me know and I will see if I can find email thread with Symantec engineering when I had the issue in UNIX (when I was a Symantec Consultant).

I would leave a vxstat -i 60 running on both servers for a few hours at least, just to verify this is the issue, so to prove the secondary is consistently writing more than the primary and NOT that occasionly there is a burst of writes at the primary, whicj would be alternate reason why VVR does't catch up.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

mhab11's picture

Update, I have had the vxstat running on both server for about an hour, the primary has never gone above 1000 and the secondary only a couple times has gone to about 1950 otherwise it has been above 2000. I have also kept vxrlink running and it has been going up and down like seen before.

I opened a ticket and the tech is going to call me in a few mins here. I will let you know the final outcome.

mhab11's picture

Solution. Replicator log volume size was to small. Increased the size to 50GB and all blocks replicated with VVR now keeping up with the program.