Video Screencast Help

node A not showing MAC address of heart interfaces of Node B

Created: 25 Mar 2010 • Updated: 06 Oct 2010 | 10 comments

Hi All ,

We have issue that the NodeA not showing MAC address of both heartbeat interfaces of NodeB.  But there is no issues in NodeB.

Operating System: Solaris 10 X86

NodeA  - lltstat output
==================
root:NodeA# lltstat -nvv
LLT node information:
    Node                 State    Link  Status  Address
   * 0 NodeA       OPEN
                                  e1000g1   UP      00:1C:79:55:79:C9
                                  e1000g3   UP      00:1D:73:55:73:9D
                                  lowpri   UP      00:1K:73:B7:05:10
     1 NodeB       OPEN
                                  e1000g1   UP
                                  e1000g3   UP
                                  lowpri   UP      00:1C:79:C8:8A:FA
root:NodeA#

NodeB - lltstat output
=================

root:NodeB# lltstat -nvv
LLT node information:
    Node                 State    Link  Status  Address
     0 NodeA       OPEN
                                  e1000g1   UP       00:1C:79:55:79:C9
                                  e1000g3   UP      00:1D:73:55:73:9D
                                  lowpri   UP      00:1K:73:B7:05:10
   * 1 NodeB       OPEN
                                  e1000g1   UP      00:1C:79:58:6C:61
                                  e1000g3   UP      00:1C:79:58:87:F1
                                  lowpri   UP      00:1C:79:C8:8A:FA
root:NodeB#

Due the above issue, the below messages logged in /var/adm/messages on NodeA frequently for every 2 seconds.

Mar 25 09:27:40 NodeA genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (e1000g1) node 1 in trouble
Mar 25 09:27:40 NodeA genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (e1000g1) node 1 in trouble
Mar 25 09:27:42 NodeA genunix: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 0 (e1000g1) node 1 active
Mar 25 09:27:42 NodeA genunix: [ID 860062 kern.notice] LLT INFO V-14-1-10024 link 0 (e1000g1) node 1 active
Mar 25 09:27:42 NodeA genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (e1000g3) node 1 inactive 8 sec (638636)
Mar 25 09:27:42 NodeA genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (e1000g3) node 1 inactive 8 sec (638636)
Mar 25 09:27:43 NodeA genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (e1000g3) node 1 inactive 9 sec (638638)
Mar 25 09:27:43 NodeA genunix: [ID 487101 kern.notice] LLT INFO V-14-1-10032 link 1 (e1000g3) node 1 inactive 9 sec (638638)
Mar 25 09:27:44 NodeA genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (e1000g1) node 1 in trouble
Mar 25 09:27:44 NodeA genunix: [ID 140958 kern.notice] LLT INFO V-14-1-10205 link 0 (e1000g1) node 1 in trouble

Could you please advice how to resolve this..

Regards,
Rajesh.

Comments 10 CommentsJump to latest comment

vcs_man's picture

Hi Rajesh,

What is the VCS version you are using?

Refer to http://support.veritas.com/docs/247259

If the Test mentioned in the above is successful from both the nodes then will have to troubleshoot from LLT side.

If the test is not successful from either of the node, then need to check from hardware side like network cables/switch ports etc..

Also, show us /etc/llttab file from both the nodes.

Gaurav Sangamnerkar's picture

What is mac setting at OBP level ?

local_mac_address ?

Gaurav

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

grkrajesh's picture

Hi Gaurav ,

It is X86 server installed solaris.

Regards,
Rajesh.

Gaurav Sangamnerkar's picture

oh sorry, missed x86,

Can you paste /etc/llttab & /etc/llthosts file from both the nodes ?

Gaurav

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

grkrajesh's picture

Hi Gaurav ,

Please find the requested info

root:NodeA# cat /etc/llthosts
0 NodeA
1 NodeB
root:NodeA# cat /etc/llttab
set-node /etc/VRTSvcs/conf/config/sysname
set-cluster 113
link e1000g1 /dev/e1000g:1 - ether - -
link e1000g3 /dev/e1000g:3 - ether - -
link-lowpri lowpri /dev/bnx:1 - ether - -
exclude 2-31
root:NodeA#
 
Node B
======
root:NodeB# cat /etc/llthosts
0 NodeA
1 NodeB
root:NodeB# cat /etc/llttab
set-node /etc/VRTSvcs/conf/config/sysname
set-cluster 113
link e1000g1 /dev/e1000g:1 - ether - -
link e1000g3 /dev/e1000g:3 - ether - -
link-lowpri lowpri /dev/bnx:1 - ether - -
exclude 2-31
root:NodeB#

Node A
======

Regards,
Rajesh.

Gaurav Sangamnerkar's picture

hmm, don't see any issue with the LLT files..

Looks odd, just to know, have you ever attempted to restart LLT (shutdown VCS, shutdown GAB & then restart LLT) ? or else reboot node ? is the state same even after reboot ?

need more info:

# modinfo | egrep 'llt|gab'
# pkginfo -l VRTSllt
# pkgchk VRTSllt
# gabconfig -a

Also did you tried the tests in above mentioned technote ? I believe they would be successfull, interesting would be to see if getmac got the mac address...

Gaurav

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.
 

grkrajesh's picture

Hi Gaurav ,

Please find the requested output below

root:NodeA# modinfo | egrep 'llt|gab'
236 fffffffff0603000  48850 213   1  gab (GAB device 5.0)
237 fffffffff0634000  1e7f0 214   1  llt (LLT 5.0)

root:NodeA# pkginfo -l VRTSllt
   PKGINST:  VRTSllt
      NAME:  Veritas Low Latency Transport by Symantec
  CATEGORY:  system
      ARCH:  i386
   VERSION:  5.0
   BASEDIR:  /
    VENDOR:  Symantec Corporation
      DESC:  Veritas Low Latency Transport by Symantec
    PSTAMP:  Veritas-5.0-03/09/07-05:11:20
  INSTDATE:  Nov 26 2007 10:30
    STATUS:  completely installed
     FILES:       83 installed pathnames
                  21 shared pathnames
                   1 linked files
                  24 directories
                  14 executables
                2069 blocks used (approx)

root:NodeA# pkgchk VRTSllt
root:NodeA# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen  15ad82e membership 01
Port b gen  15ad830 membership 01
Port h gen  15ad831 membership 01
root:NodeA#

root:NodeB# modinfo | egrep 'llt|gab'
244 ffffffffefeaa000  48850 213   1  gab (GAB device 5.0)
245 ffffffffefedb000  1e7f0 214   1  llt (LLT 5.0)

root:NodeB# pkginfo -l VRTSllt
   PKGINST:  VRTSllt
      NAME:  Veritas Low Latency Transport by Symantec
  CATEGORY:  system
      ARCH:  i386
   VERSION:  5.0
   BASEDIR:  /
    VENDOR:  Symantec Corporation
      DESC:  Veritas Low Latency Transport by Symantec
    PSTAMP:  Veritas-5.0-03/09/07-05:11:20
  INSTDATE:  Nov 26 2007 10:27
    STATUS:  completely installed
     FILES:       83 installed pathnames
                  21 shared pathnames
                   1 linked files
                  24 directories
                  14 executables
                2069 blocks used (approx)

root:NodeB# pkgchk VRTSllt
root:NodeB# gabconfig -a
GAB Port Memberships
===============================================================
Port a gen  15ad82e membership 01
Port b gen  15ad830 membership 01
Port h gen  15ad831 membership 01
root:NodeB#root:NodeA# modinfo | egrep 'llt|gab'
236 fffffffff0603000  48850 213   1  gab (GAB device 5.0)
237 fffffffff0634000  1e7f0 214   1  llt (LLT 5.0)
root:NodeA# pkginfo -l VRTSllt
   PKGINST:  VRTSllt
      NAME:  Veritas Low Latency Transport by Symantec
  CATEGORY:  system
      ARCH:  i386
   VERSION:  5.0
   BASEDIR:  /
    VENDOR:  Symantec Corporation
      DESC:  Veritas Low Latency Transport by Symantec
    PSTAMP:  Veritas-5.0-03/09/07-05:11:20
  INSTDATE:  Nov 26 2007 10:30
    STATUS:  completely installed
     FILES:       83 installed pathnames
                  21 shared pathnames
                   1 linked files
                  24 directories
                  14 executables
                2069 blocks used (approx)

Regards,
Rajesh.

vcs_man's picture

Hi Rajesh,

I am hoping that the actual hostname(/etc/llthosts) and the name in  /etc/VRTSvcs/conf/config/sysname  is same. Which is getting resolved by any of the mechanism in your network like DNS or /etc/hosts etc...

As mentioned earlier, execute DLPIping test as per the Technote (http://support.veritas.com/docs/247259) and send us results.

Thanks,
Mandar

grkrajesh's picture

Hi Mandar ,

Please find the requested dlpiping output  from both the nodes

eroot:NodeA# /opt/VRTSllt/getmac /dev/e1000g:1
/dev/e1000g:1   00:1C:79:55:79:C9
root:NodeA# /opt/VRTSllt/dlpiping -s /dev/e1000g:1
dlpiping: dlrecv do_get failed
dlpiping: recv ECHO_REQ failed
^Croot:NodeA# /opt/VRTSllt/getmac /dev/e1000g:3
/dev/e1000g:3   00:1D:73:55:73:9D
root:NodeA# /opt/VRTSllt/dlpiping -s /dev/e1000g:3
dlpiping: dlrecv do_get failed
dlpiping: recv ECHO_REQ failed
^Croot:NodeA#

 
root:NodeB# /opt/VRTSllt/dlpiping -c /dev/e1000g:1 00:1C:79:55:79:C9
00:1C:79:55:79:C9 is alive
root:NodeB# /opt/VRTSllt/dlpiping -c /dev/e1000g:3 00:1D:73:55:73:9D
00:1D:73:55:73:9D is alive
root:NodeB#

===========================================================================================

root:NodeB# /opt/VRTSllt/getmac /dev/e1000g:1
/dev/e1000g:1   00:1C:79:58:6C:61
root:NodeB# /opt/VRTSllt/dlpiping -s /dev/e1000g:1
dlpiping: dlrecv do_get failed
dlpiping: recv ECHO_REQ failed
^Croot:NodeB# /opt/VRTSllt/getmac /dev/e1000g:3
/dev/e1000g:3   00:1C:79:58:87:F1
root:NodeB# /opt/VRTSllt/dlpiping -s /dev/e1000g:3
dlpiping: dlrecv do_get failed
dlpiping: recv ECHO_REQ failed
^Croot:NodeB#
 
root:NodeA# /opt/VRTSllt/dlpiping -c /dev/e1000g:1 00:1C:79:58:6C:61
00:1C:79:58:6C:61 is alive
root:NodeA# /opt/VRTSllt/dlpiping -c /dev/e1000g:3 00:1C:79:58:87:F1
00:1C:79:58:87:F1 is alive
root:NodeA#

Regards,
Rajesh.

Gaurav Sangamnerkar's picture

Hi, Something is not correct for sure.....

can you try other tools in same VRTSllt folder, try using lltping & see the results...

Also, hv you ever tried to reload LLT modules on both the nodes ? (equivalent to reboot) ?

Gaurav

PS: If you are happy with the answer provided, please mark the post as solution. You can do so by clicking link "Mark as Solution" below the answer provided.