blade servers form SFRAC cluster, LLT NIC working mode Auto-Negotiation often cause "network partition" and lead node panic

Article:TECH188050  |  Created: 2012-05-03  |  Updated: 2012-09-22  |  Article URL http://www.symantec.com/docs/TECH188050
Article Type
Technical Solution


Environment

Issue



blade servers form SFRAC cluster, LLT NIC working mode Auto-Negotiation often cause "network partition" and lead node panic


Error



KERNEL: usr/lib/debug/lib/modules/2.6.18-164.el5/vmlinux  
   DUMPFILE: /var/crash/2012-04-24-18:48/vmcore  
       CPUS: 16  
       DATE: Tue Apr 24 18:44:47 2012  
     UPTIME: 00:24:12  
LOAD AVERAGE: 47.97, 17.43, 7.23  
      TASKS: 769  
   NODENAME: SBCJems2  
    RELEASE: 2.6.18-164.el5  
    VERSION: #1 SMP Fri Dec 3 08:56:42 CST 2010  
    MACHINE: x86_64  (2133 Mhz)  
     MEMORY: 23.6 GB  
      PANIC: "Kernel panic - not syncing: GAB: Port d halting system due to network failure at [14:2027]"  
        PID: 7691  
    COMMAND: "lltdlv"  
       TASK: ffff810314e0e860  [THREAD_INFO: ffff810314d6a000]  
        CPU: 11  
      STATE: TASK_RUNNING (PANIC)

     KERNEL: usr/lib/debug/lib/modules/2.6.18-164.el5/vmlinux  
   DUMPFILE: /var/crash/2012-04-23-07:20/vmcore  
       CPUS: 16  
       DATE: Mon Apr 23 07:15:39 2012  
     UPTIME: 3 days, 18:15:58  
LOAD AVERAGE: 105.29, 32.28, 12.47  
      TASKS: 1808  
   NODENAME: SBCJems2  
    RELEASE: 2.6.18-164.el5  
    VERSION: #1 SMP Fri Dec 3 08:56:42 CST 2010  
    MACHINE: x86_64  (2133 Mhz)  
     MEMORY: 23.6 GB  
      PANIC: "Kernel panic - not syncing: GAB: Port f halting system due to network failure at [14:2027]"  
        PID: 7889  
    COMMAND: "lltdlv"  
       TASK: ffff8104f819e040  [THREAD_INFO: ffff8104f61a0000]  
        CPU: 9  
      STATE: TASK_RUNNING (PANIC)

 


Environment



RHEL 5

SFRAC 5.1SP1


Cause



on blade server , all NIC connect through blade enclosure mainboard ,default working mode is Auto-Negotiation .

it should cause LLT NIC communication disconnect <--> connect  frequently

after kernel port rejoin to gab,  generation number conflict ,so I/O Fence panic the node


Solution



config all LLT NIC working mode to permanent mode , such as 100M Full Duplex , not using Auto-Negotiation 




Article URL http://www.symantec.com/docs/TECH188050


Terms of use for this information are found in Legal Notices