Video Screencast Help

VVR replication stopped

Created: 19 Dec 2013 • Updated: 14 Jan 2014 | 16 comments
This issue has been solved. See solution.

                   Please help me to solve this situation

replication was stopped. Problem seems to started today morning at 8:00am, due to error in writing data
in 3pp volume of ossrvg of . This caused 3pp volume detachment, which caused the replication to stop with message:

Dec 18 08:00:03  vxio: [ID 494588 kern.notice] NOTICE: VxVM VVR vxio V-5-0-602 Disconnecting rlink to_pallini_ossrvg on rvg
ossrvg: Secondary volume(s) stopped.

atmas1o dmesg
 
Dec 18 08:00:04 atmas1o vxio: [ID 467569 kern.warning] WARNING: VxVM VVR vxio V-5-0-1406 Node 10.40.240.50 disconnected from node 10.40.240.52
Dec 18 08:00:06 atmas1o vxio: [ID 467919 kern.warning] WARNING: VxVM VVR vxio V-5-0-266 Rlink to_metamorphosi_ossrvg disconnected from remote
Dec 18 08:00:06 atmas1o inetd[2436]: [ID 317013 daemon.notice] shell[23328] from 10.40.9.109 820
Dec 18 08:00:07 atmas1o inetd[2436]: [ID 317013 daemon.notice] shell[23448] from 10.40.9.109 817
Dec 18 08:00:07 atmas1o inetd[2436]: [ID 317013 daemon.notice] shell[23548] from 10.40.9.109 814
Dec 18 08:00:09 atmas1o vxio: [ID 584567 kern.notice] NOTICE: VxVM VVR vxio V-5-0-1402 Connected from node 10.40.240.50 to node 10.40.240.52
Dec 18 08:00:09 atmas1o vxio: [ID 910032 kern.notice] NOTICE: VxVM VVR vxio V-5-0-330 Unable to connect to rlink to_metamorphosi_ossrvg on rvg ossrvg: Secondary volume(s) stopped on remote
Dec 18 08:00:10 atmas1o inetd[2436]: [ID 317013 daemon.notice] shell[24521] from 10.40.9.109 811
 
 
atmas2o dmesg
 
Dec 18 08:00:02 atmas2o vxio: [ID 222684 kern.warning] WARNING: VxVM VVR vxio V-5-0-0 Volume 3pp block 12116624: Uncorrectable write error
Dec 18 08:00:02 atmas2o vxio: [ID 222684 kern.warning] WARNING: VxVM VVR vxio V-5-0-0 Volume 3pp block 4321392: Uncorrectable write error
Dec 18 08:00:02 atmas2o vxio: [ID 222684 kern.warning] WARNING: VxVM VVR vxio V-5-0-0 Volume 3pp block 6668080: Uncorrectable write error
Dec 18 08:00:03 atmas2o vxio: [ID 494588 kern.notice] NOTICE: VxVM VVR vxio V-5-0-602 Disconnecting rlink to_pallini_ossrvg on rvg ossrvg: Secondary volume(s) stopped
Dec 18 08:00:03 atmas2o last message repeated 1 time
Dec 18 08:00:03 atmas2o vxio: [ID 467919 kern.warning] WARNING: VxVM VVR vxio V-5-0-266 Rlink to_pallini_ossrvg disconnected from remote
Dec 18 08:00:07 atmas2o vxio: [ID 467569 kern.warning] WARNING: VxVM VVR vxio V-5-0-1406 Node 10.40.240.52 disconnected from node 10.40.240.50
Dec 18 08:00:10 atmas2o vxio: [ID 584567 kern.notice] NOTICE: VxVM VVR vxio V-5-0-1402 Connected from node 10.40.240.52 to node 10.40.240.50
 
 
 
atmas2o{root} #: vxprint -thf | grep -i 3pp
v 3pp ossrvg DETACHED DETACH 51200000 SELECT - fsgen
pl 3pp-01 3pp ENABLED ACTIVE 51200000 CONCAT - RW
sd disk2-26 3pp-01 disk2 1367605248 51200000 0 c1t40d0 ENA
pl 3pp-02 3pp ENABLED ACTIVE 51200000 CONCAT - RW
sd disk2mirr-01 3pp-02 disk2mirr 0 51200000 0 c3t40d0 ENA
pl 3pp-03 3pp ENABLED ACTIVE LOGONLY CONCAT - RW
sd osssrldk1-25 3pp-03 osssrldk1 419437744 512 LOG c1t40d1 ENA
pl 3pp-04 3pp ENABLED ACTIVE LOGONLY CONCAT - RW
sd osssrldk1m-25 3pp-04 osssrldk1m 419437600 512 LOG c3t40d1 EN
 
 
partial output of iostat -E of atmas2o
 
ssd0 Soft Errors: 0 Hard Errors: 5 Transport Errors: 0
Vendor: SUN Product: StorEdge 3510 Revision: 421F Serial No:
Size: 439.62GB <439624925184 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 5 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
ssd1 Soft Errors: 0 Hard Errors: 5 Transport Errors: 5
Vendor: SUN Product: StorEdge 3510 Revision: 421F Serial No:
Size: 732.71GB <732708208640 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 5 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
ssd2 Soft Errors: 0 Hard Errors: 4 Transport Errors: 0
Vendor: SUN Product: StorEdge 3510 Revision: 421F Serial No:
Size: 439.62GB <439624925184 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 4 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
ssd3 Soft Errors: 0 Hard Errors: 4 Transport Errors: 4
Vendor: SUN Product: StorEdge 3510 Revision: 421F Serial No:
Size: 732.71GB <732708208640 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 4 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
st2 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: DAT DAT72-000 Revision: C0C0 Serial No:

root@atmas1o> vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name: pallini-ossrvg
  RVG name: ossrvg
  DG name: ossdg
  RVG state: enabled for I/O
  Data volumes: 24
  VSets: 0
  SRL name: oss_srl_vol
  SRL size: 200.00 G
  Total secondaries: 1
 
Secondary:
  Host name: metamorphosi-ossrvg
  RVG name: ossrvg
  DG name: ossdg
  Data status: inconsistent
  Replication status: paused due to network disconnection
  Current mode: asynchronous
  Logging to: SRL
  Timestamp Information: N/A
 
 

Operating Systems:

Comments 16 CommentsJump to latest comment

Gaurav Sangamnerkar's picture

Hello,

You will need to bring secondary volume 3pp to "ENABLED ACTIVE" state to bring back the replication to work. As you can see the volume is in detached state.

I would still like to see the overall status of RLINK & RVG. Paste following outputs

atmas1o# vxdisk -e list

atmas1o# vxprint -qhtg ossdg

 

atmas2o# vxdisk list

atmas2o# vxdisk -e list

atmas2o# vxprint -qthg ossdg

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

symsonu's picture

oot@atmas1o> vxdisk -e list

DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR       

c0t0d0s2     auto:SVM       -            -           SVM                  c0t0d0s2         -           

c0t1d0s2     auto:SVM       -            -           SVM                  c0t1d0s2         -           

c1t40d0s2    auto:sliced    disk2        ossdg       online               c1t40d0s2        -           

c1t40d1s2    auto:sliced    osssrldk1    ossdg       online nohotuse      c1t40d1s2        -           

c2t0d0s2     auto:SVM       -            -           SVM                  c2t0d0s2         -           

c2t1d0s2     auto:none      -            -           online invalid       c2t1d0s2         -           

c3t40d0s2    auto:sliced    disk2mirr    ossdg       online               c3t40d0s2        -           

c3t40d1s2    auto:sliced    osssrldk1m   ossdg       online nohotuse      c3t40d1s2        -           

root@atmas1o> vxprint -qhtg ossdg

dg ossdg        all          all      15000    1318259952.12.atmas2o

 

dm disk2        c1t40d0s2    auto     87039    1430083072 -

dm disk2mirr    c3t40d0s2    auto     87039    1430083072 -

dm osssrldk1    c1t40d1s2    auto     81343    858497280 NOHOTUSE

dm osssrldk1m   c3t40d1s2    auto     81343    858497280 NOHOTUSE

 

v  dbdumps      -            ENABLED  ACTIVE   122880000 ROUND    -        fsgen

pl dbdumps-01   dbdumps      ENABLED  ACTIVE   122880000 CONCAT   -        RW

sd disk2mirr-22 dbdumps-01   disk2mirr 1087408128 122880000 0     c3t40d0  ENA

pl dbdumps-02   dbdumps      ENABLED  ACTIVE   122880000 CONCAT   -        RW

sd disk2-03     dbdumps-02   disk2    76365824 122880000 0        c1t40d0  ENA

pl dbdumps-03   dbdumps      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd disk2-30     dbdumps-03   disk2    1418805248 1056   LOG       c1t40d0  ENA

 

v  swap         -            ENABLED  ACTIVE   53248000 ROUND     -        gen

pl swap-01      swap         ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd disk2mirr-30 swap-01      disk2mirr 857749504 528    LOG       c3t40d0  ENA

pl swap-02      swap         ENABLED  ACTIVE   53248000 CONCAT    -        RW

sd disk2-21     swap-02      disk2    966064128 53248000 0        c1t40d0  ENA

pl swap-03      swap         ENABLED  ACTIVE   53248000 CONCAT    -        RW

sd disk2mirr-26 swap-03      disk2mirr 1240389632 53248000 0      c3t40d0  ENA

 

rv ossrvg       1            ENABLED  ACTIVE   primary  24        oss_srl_vol 

rl to_metamorphosi_ossrvg ossrvg ENABLED FAIL  metamorphosi-ossrvg ossdg to_pallini_ossrvg

v  3pp          ossrvg       ENABLED  ACTIVE   51200000 ROUND     -        fsgen

pl 3pp-01       3pp          ENABLED  ACTIVE   51200000 CONCAT    -        RW

sd disk2mirr-01 3pp-01       disk2mirr 0       51200000 0         c3t40d0  ENA

pl 3pp-02       3pp          ENABLED  ACTIVE   51200000 CONCAT    -        RW

sd disk2-01     3pp-02       disk2    0        51200000 0         c1t40d0  ENA

pl 3pp-03       3pp          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-25 3pp-03       osssrldk1 419437600 512    LOG       c1t40d1  ENA

pl 3pp-04       3pp          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-25 3pp-04      osssrldk1m 419437600 512   LOG       c3t40d1  ENA

v  JUMP         ossrvg       ENABLED  ACTIVE   25165824 ROUND     -        fsgen

pl JUMP-01      JUMP         ENABLED  ACTIVE   25165824 CONCAT    -        RW

sd disk2mirr-02 JUMP-01      disk2mirr 51200000 25165824 0        c3t40d0  ENA

pl JUMP-02      JUMP         ENABLED  ACTIVE   25165824 CONCAT    -        RW

sd disk2-02     JUMP-02      disk2    51200000 25165824 0         c1t40d0  ENA

pl JUMP-03      JUMP         ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-02 JUMP-03      osssrldk1 419430400 416    LOG       c1t40d1  ENA

pl JUMP-04      JUMP         ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-02 JUMP-04     osssrldk1m 419430400 416   LOG       c3t40d1  ENA

v  eba_ebss     ossrvg       ENABLED  ACTIVE   10485760 ROUND     -        fsgen

pl eba_ebss-01  eba_ebss     ENABLED  ACTIVE   10485760 CONCAT    -        RW

sd disk2mirr-03 eba_ebss-01  disk2mirr 76365824 10485760 0        c3t40d0  ENA

pl eba_ebss-02  eba_ebss     ENABLED  ACTIVE   10485760 CONCAT    -        RW

sd disk2-04     eba_ebss-02  disk2    199245824 10485760 0        c1t40d0  ENA

pl eba_ebss-03  eba_ebss     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-03 eba_ebss-03  osssrldk1 419430816 192    LOG       c1t40d1  ENA

pl eba_ebss-04  eba_ebss     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-03 eba_ebss-04 osssrldk1m 419430816 192   LOG       c3t40d1  ENA

v  eba_ebsw     ossrvg       ENABLED  ACTIVE   40960    ROUND     -        fsgen

pl eba_ebsw-01  eba_ebsw     ENABLED  ACTIVE   40960    CONCAT    -        RW

sd disk2mirr-04 eba_ebsw-01  disk2mirr 86851584 40960   0         c3t40d0  ENA

pl eba_ebsw-02  eba_ebsw     ENABLED  ACTIVE   40960    CONCAT    -        RW

sd disk2-05     eba_ebsw-02  disk2    209731584 40960   0         c1t40d0  ENA

pl eba_ebsw-03  eba_ebsw     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-04 eba_ebsw-03  osssrldk1 419431008 32     LOG       c1t40d1  ENA

pl eba_ebsw-04  eba_ebsw     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-04 eba_ebsw-04 osssrldk1m 419431008 32    LOG       c3t40d1  ENA

v  eba_rede     ossrvg       ENABLED  ACTIVE   10485760 ROUND     -        fsgen

pl eba_rede-01  eba_rede     ENABLED  ACTIVE   10485760 CONCAT    -        RW

sd disk2mirr-05 eba_rede-01  disk2mirr 86892544 10485760 0        c3t40d0  ENA

pl eba_rede-02  eba_rede     ENABLED  ACTIVE   10485760 CONCAT    -        RW

sd disk2-06     eba_rede-02  disk2    209772544 10485760 0        c1t40d0  ENA

pl eba_rede-03  eba_rede     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-05 eba_rede-03  osssrldk1 419431040 192    LOG       c1t40d1  ENA

pl eba_rede-04  eba_rede     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-05 eba_rede-04 osssrldk1m 419431040 192   LOG       c3t40d1  ENA

v  eba_rsdm     ossrvg       ENABLED  ACTIVE   40960    ROUND     -        fsgen

pl eba_rsdm-01  eba_rsdm     ENABLED  ACTIVE   40960    CONCAT    -        RW

sd disk2mirr-06 eba_rsdm-01  disk2mirr 97378304 40960   0         c3t40d0  ENA

pl eba_rsdm-02  eba_rsdm     ENABLED  ACTIVE   40960    CONCAT    -        RW

sd disk2-07     eba_rsdm-02  disk2    220258304 40960   0         c1t40d0  ENA

pl eba_rsdm-03  eba_rsdm     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-06 eba_rsdm-03  osssrldk1 419431232 32     LOG       c1t40d1  ENA

pl eba_rsdm-04  eba_rsdm     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-06 eba_rsdm-04 osssrldk1m 419431232 32    LOG       c3t40d1  ENA

v  eba_rtt      ossrvg       ENABLED  ACTIVE   20971520 ROUND     -        fsgen

pl eba_rtt-01   eba_rtt      ENABLED  ACTIVE   20971520 CONCAT    -        RW

sd disk2mirr-07 eba_rtt-01   disk2mirr 97419264 20971520 0        c3t40d0  ENA

pl eba_rtt-02   eba_rtt      ENABLED  ACTIVE   20971520 CONCAT    -        RW

sd disk2-08     eba_rtt-02   disk2    220299264 20971520 0        c1t40d0  ENA

pl eba_rtt-03   eba_rtt      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-07 eba_rtt-03   osssrldk1 419431264 352    LOG       c1t40d1  ENA

pl eba_rtt-04   eba_rtt      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-07 eba_rtt-04  osssrldk1m 419431264 352   LOG       c3t40d1  ENA

v  eniq_pm_vol  ossrvg       ENABLED  ACTIVE   16005120 ROUND     -        fsgen

pl eniq_pm_vol-01 eniq_pm_vol ENABLED ACTIVE   16005120 CONCAT    -        RW

sd disk2mirr-24 eniq_pm_vol-01 disk2mirr 1232816128 7573504 0     c3t40d0  ENA

sd disk2mirr-35 eniq_pm_vol-01 disk2mirr 857752192 102400 7573504 c3t40d0  ENA

sd disk2mirr-23 eniq_pm_vol-01 disk2mirr 908063872 8329216 7675904 c3t40d0 ENA

pl eniq_pm_vol-02 eniq_pm_vol ENABLED ACTIVE   16005120 CONCAT    -        RW

sd disk2-09     eniq_pm_vol-02 disk2  241270784 7675904 0         c1t40d0  ENA

sd disk2-29     eniq_pm_vol-02 disk2  1410476032 8329216 7675904  c1t40d0  ENA

pl eniq_pm_vol-03 eniq_pm_vol ENABLED ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-08 eniq_pm_vol-03 osssrldk1 419431616 256  LOG       c1t40d1  ENA

pl eniq_pm_vol-04 eniq_pm_vol ENABLED ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-08 eniq_pm_vol-04 osssrldk1m 419431616 256 LOG      c3t40d1  ENA

v  ericsson     ossrvg       ENABLED  ACTIVE   110852096 ROUND    -        fsgen

pl ericsson-01  ericsson     ENABLED  ACTIVE   110852096 CONCAT   -        RW

sd disk2mirr-08 ericsson-01  disk2mirr 118390784 95696896 0       c3t40d0  ENA

sd disk2mirr-28 ericsson-01  disk2mirr 1293637632 15155200 95696896 c3t40d0 ENA

pl ericsson-02  ericsson     ENABLED  ACTIVE   110852096 CONCAT   -        RW

sd disk2-10     ericsson-02  disk2    248946688 95696896 0        c1t40d0  ENA

sd disk2-27     ericsson-02  disk2    1373530112 15155200 95696896 c1t40d0 ENA

pl ericsson-03  ericsson     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-09 ericsson-03  osssrldk1 419431872 512    LOG       c1t40d1  ENA

pl ericsson-04  ericsson     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-09 ericsson-04 osssrldk1m 419431872 512   LOG       c3t40d1  ENA

v  export       ossrvg       ENABLED  ACTIVE   83968000 ROUND     -        fsgen

pl export-01    export       ENABLED  ACTIVE   83968000 CONCAT    -        RW

sd disk2mirr-09 export-01    disk2mirr 214087680 83968000 0       c3t40d0  ENA

pl export-02    export       ENABLED  ACTIVE   83968000 CONCAT    -        RW

sd disk2-11     export-02    disk2    344643584 83968000 0        c1t40d0  ENA

pl export-03    export       ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-10 export-03    osssrldk1 419432384 512    LOG       c1t40d1  ENA

pl export-04    export       ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-10 export-04   osssrldk1m 419432384 512   LOG       c3t40d1  ENA

v  fmsybdata    ossrvg       ENABLED  ACTIVE   31043584 ROUND     -        fsgen

pl fmsybdata-01 fmsybdata    ENABLED  ACTIVE   31043584 CONCAT    -        RW

sd disk2mirr-10 fmsybdata-01 disk2mirr 298055680 31043584 0       c3t40d0  ENA

pl fmsybdata-02 fmsybdata    ENABLED  ACTIVE   31043584 CONCAT    -        RW

sd disk2-12     fmsybdata-02 disk2    428611584 31043584 0        c1t40d0  ENA

pl fmsybdata-03 fmsybdata    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-11 fmsybdata-03 osssrldk1 419432896 480    LOG       c1t40d1  ENA

pl fmsybdata-04 fmsybdata    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-11 fmsybdata-04 osssrldk1m 419432896 480  LOG       c3t40d1  ENA

v  fmsyblog     ossrvg       ENABLED  ACTIVE   15515648 ROUND     -        fsgen

pl fmsyblog-01  fmsyblog     ENABLED  ACTIVE   15515648 CONCAT    -        RW

sd disk2mirr-11 fmsyblog-01  disk2mirr 329099264 15515648 0       c3t40d0  ENA

pl fmsyblog-02  fmsyblog     ENABLED  ACTIVE   15515648 CONCAT    -        RW

sd disk2-13     fmsyblog-02  disk2    459655168 15515648 0        c1t40d0  ENA

pl fmsyblog-03  fmsyblog     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-12 fmsyblog-03  osssrldk1 419433376 256    LOG       c1t40d1  ENA

pl fmsyblog-04  fmsyblog     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-12 fmsyblog-04 osssrldk1m 419433376 256   LOG       c3t40d1  ENA

v  home         ossrvg       ENABLED  ACTIVE   113623040 ROUND    -        fsgen

pl home-01      home         ENABLED  ACTIVE   113623040 CONCAT   -        RW

sd disk2mirr-12 home-01      disk2mirr 344614912 102400000 0      c3t40d0  ENA

sd disk2mirr-39 home-01      disk2mirr 883487872 11223040 102400000 c3t40d0 ENA

pl home-02      home         ENABLED  ACTIVE   113623040 CONCAT   -        RW

sd disk2-14     home-02      disk2    475170816 113623040 0       c1t40d0  ENA

pl home-03      home         ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-13 home-03      osssrldk1 419433632 512    LOG       c1t40d1  ENA

pl home-04      home         ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-13 home-04     osssrldk1m 419433632 512   LOG       c3t40d1  ENA

v  pm_storage   ossrvg       ENABLED  ACTIVE   162775040 ROUND    -        fsgen

pl pm_storage-01 pm_storage  ENABLED  ACTIVE   162775040 CONCAT   -        RW

sd disk2-15     pm_storage-01 disk2   588793856 140984320 0       c1t40d0  ENA

sd disk2-28     pm_storage-01 disk2   1388685312 21790720 140984320 c1t40d0 ENA

pl pm_storage-02 pm_storage  ENABLED  ACTIVE   162775040 CONCAT   -        RW

sd disk2mirr-27 pm_storage-02 disk2mirr 857708544 40960 0         c3t40d0  ENA

sd disk2mirr-36 pm_storage-02 disk2mirr 857854592 24650240 40960  c3t40d0  ENA

sd disk2mirr-37 pm_storage-02 disk2mirr 1313789952 116293120 24691200 c3t40d0 ENA

sd disk2mirr-29 pm_storage-02 disk2mirr 1210288128 21790720 140984320 c3t40d0 ENA

pl pm_storage-03 pm_storage  ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-14 pm_storage-03 osssrldk1 419434144 512   LOG       c1t40d1  ENA

pl pm_storage-04 pm_storage  ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-14 pm_storage-04 osssrldk1m 419434144 512 LOG       c3t40d1  ENA

v  pmsybdata    ossrvg       ENABLED  ACTIVE   4481024  ROUND     -        fsgen

pl pmsybdata-01 pmsybdata    ENABLED  ACTIVE   4481024  CONCAT    -        RW

sd disk2mirr-13 pmsybdata-01 disk2mirr 447014912 4481024 0        c3t40d0  ENA

pl pmsybdata-02 pmsybdata    ENABLED  ACTIVE   4481024  CONCAT    -        RW

sd disk2-16     pmsybdata-02 disk2    729778176 4481024 0         c1t40d0  ENA

pl pmsybdata-03 pmsybdata    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-15 pmsybdata-03 osssrldk1 419434656 96     LOG       c1t40d1  ENA

pl pmsybdata-04 pmsybdata    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-15 pmsybdata-04 osssrldk1m 419434656 96   LOG       c3t40d1  ENA

v  pmsyblog     ossrvg       ENABLED  ACTIVE   1855488  ROUND     -        fsgen

pl pmsyblog-01  pmsyblog     ENABLED  ACTIVE   1855488  CONCAT    -        RW

sd disk2mirr-14 pmsyblog-01  disk2mirr 451495936 1855488 0        c3t40d0  ENA

pl pmsyblog-02  pmsyblog     ENABLED  ACTIVE   1855488  CONCAT    -        RW

sd disk2-17     pmsyblog-02  disk2    734259200 1855488 0         c1t40d0  ENA

pl pmsyblog-03  pmsyblog     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-16 pmsyblog-03  osssrldk1 419434752 32     LOG       c1t40d1  ENA

pl pmsyblog-04  pmsyblog     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-16 pmsyblog-04 osssrldk1m 419434752 32    LOG       c3t40d1  ENA

v  polled_data  ossrvg       ENABLED  ACTIVE   1024000  ROUND     -        fsgen

pl polled_data-01 polled_data ENABLED ACTIVE   1024000  CONCAT    -        RW

sd disk2-18     polled_data-01 disk2  736114688 1024000 0         c1t40d0  ENA

pl polled_data-02 polled_data ENABLED ACTIVE   1024000  CONCAT    -        RW

sd disk2mirr-25 polled_data-02 disk2mirr 857667584 40960 0        c3t40d0  ENA

sd disk2mirr-38 polled_data-02 disk2mirr 882504832 983040 40960   c3t40d0  ENA

pl polled_data-03 polled_data ENABLED ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-17 polled_data-03 osssrldk1 419434784 32   LOG       c1t40d1  ENA

pl polled_data-04 polled_data ENABLED ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-17 polled_data-04 osssrldk1m 419434784 32 LOG       c3t40d1  ENA

v  segment1     ossrvg       ENABLED  ACTIVE   40960    ROUND     -        fsgen

pl segment1-01  segment1     ENABLED  ACTIVE   40960    CONCAT    -        RW

sd disk2mirr-15 segment1-01  disk2mirr 453351424 40960  0         c3t40d0  ENA

pl segment1-02  segment1     ENABLED  ACTIVE   40960    CONCAT    -        RW

sd disk2-19     segment1-02  disk2    737138688 40960   0         c1t40d0  ENA

pl segment1-03  segment1     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-18 segment1-03  osssrldk1 419434816 32     LOG       c1t40d1  ENA

pl segment1-04  segment1     ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-18 segment1-04 osssrldk1m 419434816 32    LOG       c3t40d1  ENA

v  sgwcg        ossrvg       ENABLED  ACTIVE   228884480 ROUND    -        fsgen

pl sgwcg-01     sgwcg        ENABLED  ACTIVE   228884480 CONCAT   -        RW

sd disk2mirr-16 sgwcg-01     disk2mirr 453392384 215531520 0      c3t40d0  ENA

sd disk2mirr-40 sgwcg-01     disk2mirr 894710912 13352960 215531520 c3t40d0 ENA

pl sgwcg-02     sgwcg        ENABLED  ACTIVE   228884480 CONCAT   -        RW

sd disk2-20     sgwcg-02     disk2    737179648 228884480 0       c1t40d0  ENA

pl sgwcg-03     sgwcg        ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-19 sgwcg-03     osssrldk1 419434848 512    LOG       c1t40d1  ENA

pl sgwcg-04     sgwcg        ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-19 sgwcg-04    osssrldk1m 419434848 512   LOG       c3t40d1  ENA

v  sybdata      ossrvg       ENABLED  ACTIVE   188743680 ROUND    -        fsgen

pl sybdata-01   sybdata      ENABLED  ACTIVE   188743680 CONCAT   -        RW

sd disk2mirr-17 sybdata-01   disk2mirr 668923904 188743680 0      c3t40d0  ENA

pl sybdata-02   sybdata      ENABLED  ACTIVE   188743680 CONCAT   -        RW

sd disk2-22     sybdata-02   disk2    1019312128 188743680 0      c1t40d0  ENA

pl sybdata-03   sybdata      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-20 sybdata-03   osssrldk1 419435360 512    LOG       c1t40d1  ENA

pl sybdata-04   sybdata      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-20 sybdata-04  osssrldk1m 419435360 512   LOG       c3t40d1  ENA

v  syblog       ossrvg       ENABLED  ACTIVE   80912384 ROUND     -        fsgen

pl syblog-01    syblog       ENABLED  ACTIVE   80912384 CONCAT    -        RW

sd disk2mirr-18 syblog-01    disk2mirr 921933824 80912384 0       c3t40d0  ENA

pl syblog-02    syblog       ENABLED  ACTIVE   80912384 CONCAT    -        RW

sd disk2-23     syblog-02    disk2    1208055808 80912384 0       c1t40d0  ENA

pl syblog-03    syblog       ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-21 syblog-03    osssrldk1 419435872 512    LOG       c1t40d1  ENA

pl syblog-04    syblog       ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-21 syblog-04   osssrldk1m 419435872 512   LOG       c3t40d1  ENA

v  sybmaster    ossrvg       ENABLED  ACTIVE   12095488 ROUND     -        fsgen

pl sybmaster-01 sybmaster    ENABLED  ACTIVE   12095488 CONCAT    -        RW

sd disk2mirr-19 sybmaster-01 disk2mirr 1002846208 12095488 0      c3t40d0  ENA

pl sybmaster-02 sybmaster    ENABLED  ACTIVE   12095488 CONCAT    -        RW

sd disk2-24     sybmaster-02 disk2    1288968192 12095488 0       c1t40d0  ENA

pl sybmaster-03 sybmaster    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-22 sybmaster-03 osssrldk1 419436384 192    LOG       c1t40d1  ENA

pl sybmaster-04 sybmaster    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-22 sybmaster-04 osssrldk1m 419436384 192  LOG       c3t40d1  ENA

v  upgrade      ossrvg       ENABLED  ACTIVE   33554432 ROUND     -        fsgen

pl upgrade-01   upgrade      ENABLED  ACTIVE   33554432 CONCAT    -        RW

sd disk2mirr-20 upgrade-01   disk2mirr 1014941696 33554432 0      c3t40d0  ENA

pl upgrade-02   upgrade      ENABLED  ACTIVE   33554432 CONCAT    -        RW

sd disk2-25     upgrade-02   disk2    1301063680 33554432 0       c1t40d0  ENA

pl upgrade-03   upgrade      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-23 upgrade-03   osssrldk1 419436576 512    LOG       c1t40d1  ENA

pl upgrade-04   upgrade      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-23 upgrade-04  osssrldk1m 419436576 512   LOG       c3t40d1  ENA

v  versant      ossrvg       ENABLED  ACTIVE   38912000 ROUND     -        fsgen

pl versant-01   versant      ENABLED  ACTIVE   38912000 CONCAT    -        RW

sd disk2mirr-21 versant-01   disk2mirr 1048496128 38912000 0      c3t40d0  ENA

pl versant-02   versant      ENABLED  ACTIVE   38912000 CONCAT    -        RW

sd disk2-26     versant-02   disk2    1334618112 38912000 0       c1t40d0  ENA

pl versant-03   versant      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1-24 versant-03   osssrldk1 419437088 512    LOG       c1t40d1  ENA

pl versant-04   versant      ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd osssrldk1m-24 versant-04  osssrldk1m 419437088 512   LOG       c3t40d1  ENA

v  oss_srl_vol  ossrvg       ENABLED  ACTIVE   419430400 SELECT   -        SRL

pl oss_srl_vol-01 oss_srl_vol ENABLED ACTIVE   419430400 CONCAT   -        RW

sd osssrldk1-01 oss_srl_vol-01 osssrldk1 0     419430400 0        c1t40d1  ENA

pl oss_srl_vol-02 oss_srl_vol ENABLED ACTIVE   419430400 CONCAT   -        RW

sd osssrldk1m-01 oss_srl_vol-02 osssrldk1m 0   419430400 0        c3t40d1  ENA

root@atmas1o> ssh atmas2o

Last login: Thu Dec 19 10:09:20 2013 from 10.80.10.45

You have new mail.

DISPLAY set to atmas1o:0.0

atmas2o{root} # vxdisk list

DEVICE       TYPE            DISK         GROUP        STATUS

c0t0d0s2     auto:SVM        -            -            SVM

c0t1d0s2     auto:none       -            -            online invalid

c1t40d0s2    auto:sliced     -            -            online

c1t40d1s2    auto:sliced     -            -            online

c2t0d0s2     auto:SVM        -            -            SVM

c2t1d0s2     auto:none       -            -            online invalid

c3t40d0s2    auto:sliced     -            -            online

c3t40d1s2    auto:sliced     -            -            online

atmas2o{root} # vxdisk -e list

DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR       

c0t0d0s2     auto:SVM       -            -           SVM                  c0t0d0s2         -           

c0t1d0s2     auto:none      -            -           online invalid       c0t1d0s2         -           

c1t40d0s2    auto:sliced    -            -           online               c1t40d0s2        -           

c1t40d1s2    auto:sliced    -            -           online               c1t40d1s2        -           

c2t0d0s2     auto:SVM       -            -           SVM                  c2t0d0s2         -           

c2t1d0s2     auto:none      -            -           online invalid       c2t1d0s2         -           

c3t40d0s2    auto:sliced    -            -           online               c3t40d0s2        -           

c3t40d1s2    auto:sliced    -            -           online               c3t40d1s2        -           

atmas2o{root} # vxprint -qthg ossdg

VxVM vxprint ERROR V-5-1-582 Disk group ossdg: No such disk group

 

Gaurav Sangamnerkar's picture

Hi,

atmas2o is your DR server right ? If yes,

On atmas2o server, ossdg needs to be imported so that volumes can be started .. can you import the DG & then share the vxprint output, as none of disk is in failed state, I would expect DG to import successfully

atmas2o# vxdg import ossdg

atmas2o# vxdisk list

atmas2o# vxdisk -e list

atmas2o# vxprint -qthg ossdg

 

If all the volumes in vxprint command are showing "ENABLED ACTIVE", RLINK & RVG are also ENABLED ACTIVE, you will need to start the replication once again

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

mikebounds's picture

The rlink has been detached as in your messages as the secondary is inconsistent as shown in vradmin repstatus output (because VVR couldn't write to  3pp volume volume.  So once you have fixed 3pp volume you will need to restart replication.  You will need to detach rlink on primary first:

vxrlink -g ossdg det to_metamorphosi_ossrvg

and this should clear FAIL state on the rlink

Then start replication:

vradmin -g ossdg -a startrep ossrvg

This will attach rlinks on both sites and start replication.

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

SOLUTION
symsonu's picture

I could not be able to move forward.

if the situation remain same, will it cause issue as SRL volume become full and overlaps(though its protected by DCM logging also).

How can we check SRL volume usage?

 

 

 

mikebounds's picture

You can use:

 

vxrlink -g ossdg status to_metamorphosi_ossrvg

to show SRL and DCM usage.

Mike

 

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below

symsonu's picture

C  OssfsRep       

-- RESOURCES ONLINING
-- Group           Type            Resource             System               IState

F  OssfsRep        RVG             ossrvg               atmas2o              W_ONLINE

-- RESOURCES DISABLED
-- Group           Type            Resource            

H  OssfsRep        DiskGroup       ossdg               
H  OssfsRep        IP              ossrep_ip           
H  OssfsRep        RVG             ossrvg              

-- WAN HEARTBEAT STATE
-- Heartbeat       To                   State     

M  Icmp            pallini_cluster      ALIVE     

-- REMOTE CLUSTER STATE
-- Cluster         State     

N  pallini_cluster RUNNING   

-- REMOTE SYSTEM STATE
-- cluster:system       State                Frozen              

O  pallini_cluster:atmas1o RUNNING              0                   

-- REMOTE GROUP STATE
-- Group           cluster:system       Probed     AutoDisabled    State          

P  Oss             pallini_cluster:atmas1o Y          N               ONLINE         
P  Ossfs           pallini_cluster:atmas1o Y          N               ONLINE         
P  Sybase1         pallini_cluster:atmas1o Y          N               ONLINE         
root@atmas2o:/var/adm# hagrp -resources OssfsRep
ossdg
ossrep_ip
ossrvg

when we bring this OssfsRep service group online to bring ossdg and volume started. it comes online and then again faulted

2014/01/02 14:03:37 VCS INFO V-16-1-50135 User root fired command: hagrp -online OssfsRep  atmas2o  localclus  from ::ffff:10.43.20.131
2014/01/02 14:03:37 VCS NOTICE V-16-1-10166 Initiating manual online of group OssfsRep on system atmas2o
2014/01/02 14:03:37 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group OssfsRep on all nodes
2014/01/02 14:03:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource ossdg (Owner: Unspecified, Group: OssfsRep) on System atmas2o
2014/01/02 14:03:37 VCS NOTICE V-16-1-10301 Initiating Online of Resource ossrep_ip (Owner: Unspecified, Group: OssfsRep) on System atmas2o
2014/01/02 14:03:38 VCS WARNING V-16-10001-1014 (atmas2o) DiskGroup:ossdg:online:Diskgroups will be imported without reservations
2014/01/02 14:03:40 VCS INFO V-16-1-10298 Resource ossrep_ip (Owner: Unspecified, Group: OssfsRep) is online on atmas2o (VCS initiated)
2014/01/02 14:03:40 VCS NOTICE V-16-10001-1009 (atmas2o) DiskGroup:ossdg:online:vxdg import succeeded on Disk Group ossdg
2014/01/02 14:03:40 VCS NOTICE V-16-10001-1092 (atmas2o) DiskGroup:ossdg:online:Volumes in Disk Group ossdg are started automatically as part of import command, the system level autostartvolumes flag is set to on
2014/01/02 14:03:42 VCS INFO V-16-1-10298 Resource ossdg (Owner: Unspecified, Group: OssfsRep) is online on atmas2o (VCS initiated)
2014/01/02 14:03:42 VCS NOTICE V-16-1-10301 Initiating Online of Resource ossrvg (Owner: Unspecified, Group: OssfsRep) on System atmas2o
2014/01/02 14:03:43 VCS WARNING V-16-20010-208 (atmas2o) RVG:ossrvg:online:Cannot recover Rvg ossrvg in diskgroup ossdg. Retrying...
2014/01/02 14:03:43 VCS WARNING V-16-20010-208 (atmas2o) RVG:ossrvg:online:Cannot recover Rvg ossrvg in diskgroup ossdg. Retrying...
2014/01/02 14:03:43 VCS WARNING V-16-20010-208 (atmas2o) RVG:ossrvg:online:Cannot recover Rvg ossrvg in diskgroup ossdg. Retrying...
2014/01/02 14:03:43 VCS WARNING V-16-20010-208 (atmas2o) RVG:ossrvg:online:Cannot recover Rvg ossrvg in diskgroup ossdg. Retrying...
2014/01/02 14:03:43 VCS WARNING V-16-20010-208 (atmas2o) RVG:ossrvg:online:Cannot recover Rvg ossrvg in diskgroup ossdg. Retrying...
2014/01/02 14:03:43 VCS ERROR V-16-20010-209 (atmas2o) RVG:ossrvg:online:Cannot recover Rvg ossrvg in diskgroup ossdg after 5 tries
2014/01/02 14:03:44 VCS INFO V-16-2-13716 (atmas2o) Resource(ossrvg): Output of the completed operation (online)
==============================================
VxVM VVR vxrvg WARNING V-5-1-3227  Rvg ossrvg already recovered, but  state is FAIL
VxVM VVR vxrvg WARNING V-5-1-3227  Rvg ossrvg already recovered, but  state is FAIL
VxVM VVR vxrvg WARNING V-5-1-3227  Rvg ossrvg already recovered, but  state is FAIL
VxVM VVR vxrvg WARNING V-5-1-3227  Rvg ossrvg already recovered, but  state is FAIL
VxVM VVR vxrvg WARNING V-5-1-3227  Rvg ossrvg already recovered, but  state is FAIL

I think , its due to the fact that
root@atmas2o:/var/adm# vxprint -ht | grep "^r"
rv ossrvg       1            DISABLED FAIL     secondary 24       oss_srl_vol  
rl to_pallini_ossrvg ossrvg  ENABLED  FAIL     pallini-ossrvg ossdg to_metamorphosi_ossrvg
root@atmas2o:/var/adm#

and rvg is disabled as 3pp in detach.

v  3pp          ossrvg       DETACHED DETACH   51200000 SELECT    -        fsgen
pl 3pp-01       3pp          DISABLED ACTIVE   51200000 CONCAT    -        RW
sd disk2-26     3pp-01       disk2    1367605248 51200000 0       c1t40d0  ENA
pl 3pp-02       3pp          DISABLED ACTIVE   51200000 CONCAT    -        RW
sd disk2mirr-01 3pp-02       disk2mirr 0       51200000 0         c3t40d0  ENA
pl 3pp-03       3pp          DISABLED(SPARSE) ACTIVE LOGONLY CONCAT -      RW
sd osssrldk1-25 3pp-03       osssrldk1 419437744 512    LOG       c1t40d1  ENA
pl 3pp-04       3pp          DISABLED(SPARSE) ACTIVE LOGONLY CONCAT -      RW
sd osssrldk1m-25 3pp-04      osssrldk1m 419437600 512   LOG       c3t40d1  ENA
v  oss_srl_vol  ossrvg       ENABLED  ACTIVE   419430400 SELECT   -        SRL
pl oss_srl_vol-01 oss_srl_vol ENABLED ACTIVE   419430400 CONCAT   -        RW
sd osssrldk1-01 oss_srl_vol-01 osssrldk1 0     419430400 0        c1t40d1  ENA
pl oss_srl_vol-02 oss_srl_vol ENABLED ACTIVE   419430400 CONCAT   -        RW
sd osssrldk1m-01 oss_srl_vol-02 osssrldk1m 0   419430400 0        c3t40d1  ENA

and there is one volume in needsync

v  sybmaster    ossrvg       ENABLED  NEEDSYNC 12095488 SELECT    -        fsgen
pl sybmaster-01 sybmaster    ENABLED  ACTIVE   12095488 CONCAT    -        RW
sd disk2-23     sybmaster-01 disk2    1283043328 12095488 0       c1t40d0  ENA
pl sybmaster-02 sybmaster    ENABLED  ACTIVE   12095488 CONCAT    -        RW
sd disk2mirr-22 sybmaster-02 disk2mirr 792752128 12095488 0       c3t40d0  ENA
pl sybmaster-03 sybmaster    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
sd osssrldk1-09 sybmaster-03 osssrldk1 419434848 192    LOG       c1t40d1  ENA
pl sybmaster-04 sybmaster    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
sd osssrldk1m-22 sybmaster-04 osssrldk1m 419436384 192  LOG       c3t40d1  ENA

I have freezed service group OSSfsRep and now i think we need to correct 3pp and sybmaster

Am I on correct path and how to correct 3pp?
Is it by deleting from RVG and again creating ?

symsonu's picture

Now , we only need to recover rlink that is in connect fail state

 

tmas1o{root} # vxprint -qhtg ossdg | egrep '^rv|^rl'
rv ossrvg       1            ENABLED  ACTIVE   primary  24        oss_srl_vol  
rl to_metamorphosi_ossrvg ossrvg CONNECT FAIL  metamorphosi-ossrvg ossdg to_pallini_ossrvg
atmas1o{root} # vradmin -g ossdg -l printrvg ossrvg
Replicated Data Set: ossrvg
Primary:
        HostName: pallini-ossrvg        <localhost>
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_metamorphosi_ossrvg, detached=off, synchronous=off
Secondary:
        HostName: metamorphosi-ossrvg
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_pallini_ossrvg, detached=off, synchronous=off

atmas1o{root} # vx^H^H^H^C
atmas1o{root} # vradmin -g ossdg -l repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  pallini-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               24
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   200.00 G
  Total secondaries:          1

Secondary:
  Host name:                  metamorphosi-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Rlink from Primary:         to_metamorphosi_ossrvg
  Rlink to Primary:           to_pallini_ossrvg
  Configured mode:            asynchronous
  Latency protection:         off
  SRL protection:             autodcm
  Data status:                inconsistent
  Replication status:         logging to DCM (needs dcm resynchronization)
  Current mode:               asynchronous
  Logging to:                 DCM (contains 0 Kbytes) (SRL protection logging)
  Timestamp Information:      N/A
  Bandwidth Limit:            N/A

atmas1o{root} # vradmin -g ossdg -l repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  pallini-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               24
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   200.00 G
  Total secondaries:          1

Secondary:
  Host name:                  metamorphosi-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Rlink from Primary:         to_metamorphosi_ossrvg
  Rlink to Primary:           to_pallini_ossrvg
  Configured mode:            asynchronous
  Latency protection:         off
  SRL protection:             autodcm
  Data status:                inconsistent
  Replication status:         logging to DCM (needs dcm resynchronization)
  Current mode:               asynchronous
  Logging to:                 DCM (contains 0 Kbytes) (SRL protection logging)
  Timestamp Information:      N/A
  Bandwidth Limit:            N/A

atmas1o{root} # ssh atmas2o
Last login: Thu Jan  2 16:22:39 2014 from atmas1o
You have new mail.
DISPLAY set to atmas1o:0.0
atmas2o{root} # vxprint -Pl
Disk group: ossdg

Rlink:    to_pallini_ossrvg
info:     timeout=500 packet_size=1452 rid=0.1279
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=FAIL
          synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=ossrvg
          remote_host=pallini-ossrvg IP_addr=10.40.240.50 port=4145
          remote_dg=ossdg
          remote_dg_dgid=1318259952.12.atmas2o
          remote_rvg_version=30
          remote_rlink=to_metamorphosi_ossrvg
          remote_rlink_rid=0.177629
          local_host=metamorphosi-ossrvg IP_addr=10.40.240.52 port=4145
protocol: UDP/IP
flags:    write enabled attached inconsistent cant_sync fail connected

atmas2o{root} # vradmin -g ossdg -l printrvg ossrvg
Replicated Data Set: ossrvg
Primary:
        HostName: pallini-ossrvg
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_metamorphosi_ossrvg, detached=off, synchronous=off
Secondary:
        HostName: metamorphosi-ossrvg   <localhost>
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_pallini_ossrvg, detached=off, synchronous=off

atmas2o{root} # vradmin -g ossdg -l repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  pallini-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               24
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   200.00 G
  Total secondaries:          1

Secondary:
  Host name:                  metamorphosi-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Rlink from Primary:         to_metamorphosi_ossrvg
  Rlink to Primary:           to_pallini_ossrvg
  Configured mode:            asynchronous
  Latency protection:         off
  SRL protection:             autodcm
  Data status:                inconsistent
  Replication status:         logging to DCM (needs dcm resynchronization)
  Current mode:               asynchronous
  Logging to:                 DCM (contains 0 Kbytes) (SRL protection logging)
  Timestamp Information:      N/A
  Bandwidth Limit:            N/A

atmas2o{root} #
 

atmas2o{root} # vxprint -htqg ossdg | grep "^r"
rv ossrvg       1            ENABLED  ACTIVE   secondary 24       oss_srl_vol  
rl to_pallini_ossrvg ossrvg  CONNECT  FAIL     pallini-ossrvg ossdg to_metamorphosi_ossrvg
atmas2o{root}

Gaurav Sangamnerkar's picture

Hi,

I would recommend to stop & start the replication again ... refer

https://www-secure.symantec.com/connect/articles/r...

 

Stop Replication

-bash-3.2$ sudo vradmin -g diskgroup -f stoprep home-rvg

VxVM VVR vradmin WARNING V-5-52-285  Secondary or Bunker will become out-of-date.

vradmin: Continue with stoprep (y/n)? y

Message from Primary:

VxVM VVR vxrlink INFO V-5-1-6466 Data volumes are in use. Before restarting replication a complete synchronization of the secondary data volumes must be performed.

Start Replication

-bash-3.2$ sudo vradmin -g diskgroup -a startrep home-rvg

Message from Primary:

VxVM VVR vxrlink WARNING V-5-1-3359 Attaching rlink to non-empty rvg. Autosync will be performed.

VxVM VVR vxrlink INFO V-5-1-3614 Secondary data volumes detected with rvg home-rvg as parent:

VxVM VVR vxrlink INFO V-5-1-6183 home:         len=624951296            primary_datavol=home

VxVM VVR vxrlink INFO V-5-1-3365 Autosync operation has started

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

symsonu's picture

hello gaurav

 

please have a look at below. i executed the action plan

 

vradmin -g ossdg -f stoprep ossrvg

 

vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name: pallini-ossrvg
  RVG name: ossrvg
  DG name: ossdg
  RVG state: enabled for I/O
  Data volumes: 24
  VSets: 0
  SRL name: oss_srl_vol
  SRL size: 200.00 G
  Total secondaries: 1

Secondary:
  Host name: metamorphosi-ossrvg
  RVG name: ossrvg
  DG name: ossdg
  Data status: inconsistent
  Replication status: stopped (primary detached)
  Current mode: N/A
  Logging to: SRL
  Timestamp Information: N/A

vradmin -g ossdg -f startrep ossrvg
Message from Primary:
VxVM VVR vxrlink WARNING V-5-1-12397 This command should only be used if primary and all secondaries are already synchronized. If this is not the case detach the rlink and use autosync or checkpoint options to attach.
VxVM VVR vxrlink INFO V-5-1-3614 Secondary data volumes detected with rvg ossrvg as parent:
VxVM VVR vxrlink INFO V-5-1-6183 3pp: len=51200000 primary_datavol=3pp
VxVM VVR vxrlink INFO V-5-1-6183 JUMP: len=25165824 primary_datavol=JUMP
VxVM VVR vxrlink INFO V-5-1-6183 eba_ebss: len=10485760 primary_datavol=eba_ebss
VxVM VVR vxrlink INFO V-5-1-6183 eba_ebsw: len=40960 primary_datavol=eba_ebsw
VxVM VVR vxrlink INFO V-5-1-6183 eba_rede: len=10485760 primary_datavol=eba_rede
VxVM VVR vxrlink INFO V-5-1-6183 eba_rsdm: len=40960 primary_datavol=eba_rsdm
VxVM VVR vxrlink INFO V-5-1-6183 eba_rtt: len=20971520 primary_datavol=eba_rtt
VxVM VVR vxrlink INFO V-5-1-6183 eniq_pm_vol: len=16005120 primary_datavol=eniq_pm_vol
VxVM VVR vxrlink INFO V-5-1-6183 ericsson: len=110852096 primary_datavol=ericsson
VxVM VVR vxrlink INFO V-5-1-6183 export: len=83968000 primary_datavol=export
VxVM VVR vxrlink INFO V-5-1-6183 fmsybdata: len=31043584 primary_datavol=fmsybdata
VxVM VVR vxrlink INFO V-5-1-6183 fmsyblog: len=15515648 primary_datavol=fmsyblog
VxVM VVR vxrlink INFO V-5-1-6183 home: len=113623040 primary_datavol=home
VxVM VVR vxrlink INFO V-5-1-6183 pm_storage: len=162775040 primary_datavol=pm_storage
VxVM VVR vxrlink INFO V-5-1-6183 pmsybdata: len=4481024 primary_datavol=pmsybdata
VxVM VVR vxrlink INFO V-5-1-6183 pmsyblog: len=1855488 primary_datavol=pmsyblog
VxVM VVR vxrlink INFO V-5-1-6183 polled_data: len=1024000 primary_datavol=polled_data
VxVM VVR vxrlink INFO V-5-1-6183 segment1: len=40960 primary_datavol=segment1
VxVM VVR vxrlink INFO V-5-1-6183 sgwcg: len=228884480 primary_datavol=sgwcg
VxVM VVR vxrlink INFO V-5-1-6183 sybdata: len=188743680 primary_datavol=sybdata
VxVM VVR vxrlink INFO V-5-1-6183 syblog: len=80912384 primary_datavol=syblog
VxVM VVR vxrlink INFO V-5-1-6183 sybmaster: len=12095488 primary_datavol=sybmaster
VxVM VVR vxrlink INFO V-5-1-6183 upgrade: len=33554432 primary_datavol=upgrade
VxVM VVR vxrlink INFO V-5-1-6183 versant: len=38912000 primary_datavol=versant
 

Gaurav Sangamnerkar's picture

You are missing with "-a" switch in the startrep command which will trigger the autosync

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

symsonu's picture

 

I executed  command

vradmin -g ossdg -f startrep ossrvg

and it showed me all up todate in few seconds. when the system was oyt of replication from last 20 days.

 

Gaurav Sangamnerkar's picture

You have forced started replication without autosync which has marked replication upto date, secondary site may not be upto date, stop the replication & start again with startrep command however add "-a" as indicated in above posts .. for e.g

-bash-3.2$ sudo vradmin -g <diskgroup> -a startrep <rvg>

 

G

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

symsonu's picture

I did the full sync

now status is as below

 

root@atmas1o:/ericsson/hagcs/log# vradmin -l printrvg
Replicated Data Set: ossrvg
Primary:
        HostName: pallini-ossrvg        <localhost>
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_metamorphosi_ossrvg, detached=on, synchronous=off
Secondary:
        HostName: metamorphosi-ossrvg
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_pallini_ossrvg, detached=on, synchronous=off
Secondary:
        HostName: metamorphosi-ossrvg
        RvgName: ossrvg
        DgName: ossdg
        datavol_cnt: 24
        vset_cnt: 0
        srl: oss_srl_vol
        RLinks:
            name=to_pallini_ossrvg, detached=on, synchronous=off
Config Errors:
        metamorphosi-ossrvg: unknown

 

 

 

root@atmas1o:/ericsson/hagcs/log# vradmin -g ossdg repstatus ossrvg
Replicated Data Set: ossrvg
Primary:
  Host name:                  pallini-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  RVG state:                  enabled for I/O
  Data volumes:               24
  VSets:                      0
  SRL name:                   oss_srl_vol
  SRL size:                   200.00 G
  Total secondaries:          1

Secondary:
  Host name:                  metamorphosi-ossrvg
  RVG name:                   ossrvg
  DG name:                    ossdg
  Data status:                inconsistent
  Replication status:         stopped (primary detached)
  Current mode:               N/A
  Logging to:                 SRL
  Timestamp Information:      N/A

Config Errors:
  metamorphosi-ossrvg:        unknown

root@atmas1o:/ericsson/hagcs/log#

 

 

root@atmas1o:/ericsson/hagcs/log# vxrlink -g ossdg status to_metamorphosi_ossrvg
VxVM VVR vxrlink INFO V-5-1-3539 Rlink to_metamorphosi_ossrvg is not currently attached (STALE).
You have new mail in /var/mail/root
root@atmas1o:/ericsson/hagcs/log#

 

 

Now if i start replication with below command, will there be any impact, as the primary is not buffering data as this status is there from last night.

vradmin -g ossdg -s startrep ossrvg

 

 

mikebounds's picture

Your rlink is attached so VVR is not keeping track of any writes.

You need to run "vradmin -g ossdg -a startrep ossrvg" (with -a, not -s) to establish replication and then VVR will sync the secondary over the network.

If this command throws back an error, then please provide output of

vxprint -VPl

 

from both nodes

Mike

UK Symantec Consultant in VCS, GCO, SF, VVR, VxAT on Solaris, AIX, HP-ux, Linux & Windows

If this post has answered your question then please click on "Mark as solution" link below