Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

ODM on RHEL 5.5 with SFHA 5.1 SP1 RP2 (oracle 10g)

Created: 17 Oct 2012 • Updated: 04 Dec 2012 | 2 comments
This issue has been solved. See solution.

Hi to all,

we are trying to use ODM for Oracle Database Clustered  10g (NO RAC) with VCS 5.1 on RHEL 5.5 64bit.

Referencing Symatec document Veritas Storage Foundation: Storage and Availability Management for Oracle and Databases for Linux 5.1.

After ODM setup the Oracle Database Startup report in the alert about the correct usage of ODM but as soon as we try to create new tablespace

on the shared storage the execution hangs and causing disconnect  the "oracle user".

in the /var/log/messages:

 

Oct 17 10:40:31 narepssg1 kernel: LLT INFO V-14-1-10541 llt_send_hb: timer not called for 2 secs (2289 ticks). Send out of context h
bs to peers from llt_eth_recv. 177 secs more to go
Oct 17 10:40:33 narepssg1 kernel: LLT INFO V-14-1-10541 llt_send_hb: timer not called for 4 secs (4338 ticks). Send out of context h
bs to peers from llt_eth_recv. 175 secs more to go
Oct 17 10:40:35 narepssg1 kernel: LLT INFO V-14-1-10541 llt_send_hb: timer not called for 6 secs (6388 ticks). Send out of context h
bs to peers from llt_eth_recv. 173 secs more to go
Oct 17 10:40:37 narepssg1 kernel: LLT INFO V-14-1-10541 llt_send_hb: timer not called for 8 secs (8438 ticks). Send out of context h
bs to peers from llt_eth_recv. 171 secs more to go
Oct 17 10:40:38 narepssg1 kernel: BUG: soft lockup - CPU#25 stuck for 10s! [ksoftirqd/25:78]
Oct 17 10:40:38 narepssg1 kernel: CPU 25:
Oct 17 10:40:38 narepssg1 kernel: Modules linked in: vxodm(PFU) vxfen(PU) gab(PU) llt(PU) autofs4 hidp rfcomm l2cap bluetooth dmpjbo
d(PU) dmpap(PU) dmpaa(PU) vxspec(PFU) vxio(PFU) vxdmp(PU) lockd sunrpc cpufreq_ondemand acpi_cpufreq freq_table bonding vxportal(PFU
) fdd(PFU) vxfs(PU) exportfs dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button ba
ttery asus_acpi acpi_memhotplug ac ipv6 xfrm_nalgo crypto_api parport_pc lp parport joydev sr_mod ide_cd igb cdrom i2c_i801 cdc_ethe
r 8021q usbnet pcspkr i2c_core bnx2 dca dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod
 mppVhba(U) usb_storage qla2xxx scsi_transport_fc ata_piix libata shpchp megaraid_sas mppUpper(U) sg sd_mod scsi_mod ext3 jbd uhci_h
cd ohci_hcd ehci_hcd
Oct 17 10:40:38 narepssg1 kernel: Pid: 78, comm: ksoftirqd/25 Tainted: PF     2.6.18-194.el5 #1
Oct 17 10:40:38 narepssg1 kernel: RIP: 0010:[<ffffffff80065bfc>]  [<ffffffff80065bfc>] .text.lock.spinlock+0x2/0x30
Oct 17 10:40:38 narepssg1 kernel: RSP: 0018:ffff81027fd83ab0  EFLAGS: 00000282
Oct 17 10:40:38 narepssg1 kernel: RAX: 0000000000000041 RBX: ffff810330bfe080 RCX: 0000000000000001
Oct 17 10:40:38 narepssg1 kernel: RDX: 0000000000000040 RSI: 000000000fbd4c18 RDI: ffff81037b908b5c
Oct 17 10:40:38 narepssg1 kernel: RBP: ffff81027fd83a30 R08: 0000000000000202 R09: ffff81027f75f000
Oct 17 10:40:38 narepssg1 kernel: R10: ffff8103441f2540 R11: ffffffff88738855 R12: ffffffff8005ec8e
Oct 17 10:40:38 narepssg1 kernel: R13: ffff81047d58d7e8 R14: ffffffff8007922b R15: ffff81027fd83a30
Oct 17 10:40:38 narepssg1 kernel: FS:  0000000000000000(0000) GS:ffff81047fed2840(0000) knlGS:0000000000000000
Oct 17 10:40:38 narepssg1 kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Oct 17 10:40:38 narepssg1 kernel: CR2: 00000034f80c6060 CR3: 00000001680d7000 CR4: 00000000000006e0
Oct 17 10:40:38 narepssg1 kernel:
Oct 17 10:40:38 narepssg1 kernel: Call Trace:
Oct 17 10:40:38 narepssg1 kernel:  <IRQ>  [<ffffffff88738802>] :vxfs:vx_aio_iodone+0x15c/0x1af
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff887373ca>] :vxfs:vx_dio_bio_done+0x82/0x93
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8894588a>] :vxio:volkiodone+0x204/0x22e
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff88184e48>] :qla2xxx:qla24xx_process_response_queue+0xa8/0x224
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8003ddd5>] lock_timer_base+0x1b/0x3c
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff80032652>] del_timer+0x7c/0x85
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff88078c8b>] :scsi_mod:scsi_delete_timer+0x12/0x59
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8008db07>] enqueue_task+0x41/0x56
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8008db72>] __activate_task+0x56/0x6d
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff800dce08>] cache_flusharray+0x2f/0xa3
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8896990f>] :vxio:volsiodone+0x488/0x4d2
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8896c5d6>] :vxio:vol_subdisksio_done+0x98/0xa3
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff88943778>] :vxio:volkcontext_process+0x75/0x136
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff88934564>] :vxio:voldiskiodone+0x23a/0x289
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff88934cf7>] :vxio:voldiskiodone_intr+0x126/0x12f
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8892eef0>] :vxio:volsp_iodone_common+0x10d/0x114
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8002cfe9>] __end_that_request_first+0x23c/0x5bf
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8005c78e>] blk_run_queue+0x28/0x73
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff88079fe5>] :scsi_mod:scsi_end_request+0x27/0xcd
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8807a1d9>] :scsi_mod:scsi_io_completion+0x14e/0x324
Oct 17 10:40:38 narepssg1 kernel:  [<ffffffff8807a1d9>] :scsi_mod:scsi_io_completion+0x14e/0x324
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff880a7802>] :sd_mod:sd_rw_intr+0x252/0x28c
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff8807a46e>] :scsi_mod:scsi_device_unbusy+0x67/0x81
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff80037fd5>] blk_done_softirq+0x5f/0x6d
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff80012409>] __do_softirq+0x89/0x133
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff8005f2fc>] call_softirq+0x1c/0x28
Oct 17 10:40:39 narepssg1 kernel:  <EOI>  [<ffffffff8009609a>] ksoftirqd+0x0/0xbf
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff8006dba8>] do_softirq+0x2c/0x85
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff800960f9>] ksoftirqd+0x5f/0xbf
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff80032bdc>] kthread+0xfe/0x132
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff8005efb1>] child_rip+0xa/0x11
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff80032ade>] kthread+0x0/0x132
Oct 17 10:40:39 narepssg1 kernel:  [<ffffffff8005efa7>] child_rip+0x0/0x11

 

So, could you help me about that ?
thanks in advance,
Vincenzo

 

 

 

 

 

Comments 2 CommentsJump to latest comment

Yasuhisa Ishikawa's picture

After ODM setup the Oracle Database Startup report in the alert about the correct usage of ODM

How did you get reported?
Have you already open a support case?

Call trace show us the kernel is stacked in scsi_mod - it is hard to what's wrong only with this trace, but scsi_mod is one of suspects.
It is better to open a support case with Symantec and RedHat ASAP.

Authorized Symantec Consultant(ASC) Data Protection in Tokyo, Japan

Ed Menze's picture

Hi, Vincenzo,

You've hit a known bug (2726056) in ODM/VxFS 5.1sp1rp2, which is resolved in rp3.

Regards,

Ed

 

 

SOLUTION