RT Self Service / #131747: Kernel crash linux-3.6.11 openafs-1.6.2

Hello,
today we had a crash with a machine running linux kernel 3.6.11 and
openafs 1.6.2:

Oct 16 14:25:03 lnx-16 kernel: afs: disk cache read error in CacheItems slot 106713 off 8537060/9303540 code -4/80
Oct 16 14:25:03 lnx-16 kernel: openafs: assertion failed: tdc, file: /tmp/S/openafs-1.6.2/src/libafs/MODLOAD-3.6.11-SP/afs_dca
che.c, line: 1512
Oct 16 14:25:03 lnx-16 kernel: ------------[ cut here ]------------
Oct 16 14:25:03 lnx-16 kernel: kernel BUG at /tmp/S/openafs-1.6.2/src/libafs/MODLOAD-3.6.11-SP/afs_dcache.c:1512!
Oct 16 14:25:03 lnx-16 kernel: invalid opcode: 0000 [#1] PREEMPT SMP
Oct 16 14:25:03 lnx-16 kernel: Modules linked in: fuse openafs(PO) af_packet rtc ext3 jbd loop hid_generic usbhid hid sr_mod c
drom sg ehci_hcd coretemp hwmon evdev psmouse qla2xxx mpt2sas uhci_hcd raid_class ata_piix usbcore usb_common bnx2 button proc
essor unix
Oct 16 14:25:03 lnx-16 kernel: CPU 9
Oct 16 14:25:03 lnx-16 kernel: Pid: 32537, comm: ipython Tainted: P O 3.6.11 #1 Dell Inc. PowerEdge R610/0F0XJ6
Oct 16 14:25:03 lnx-16 kernel: RIP: 0010:[<ffffffffa02b6219>] [<ffffffffa02b6219>] afs_AllocDCache+0x469/0x4c0 [openafs]
Oct 16 14:25:03 lnx-16 kernel: RSP: 0018:ffff880500319708 EFLAGS: 00010292
Oct 16 14:25:03 lnx-16 kernel: RAX: 0000000000000070 RBX: 0000000000000000 RCX: 000000000000002f
Oct 16 14:25:03 lnx-16 kernel: RDX: 0000000000000044 RSI: 0000000000000046 RDI: ffffffff815233f4
Oct 16 14:25:03 lnx-16 kernel: RBP: ffff880500319748 R08: 0000000000000000 R09: 0000000000000455
Oct 16 14:25:03 lnx-16 kernel: R10: ffff8800000b86e0 R11: 000000000002a940 R12: 0000000000000000
Oct 16 14:25:03 lnx-16 kernel: R13: 0000000000000000 R14: ffff88092537f0c0 R15: 000000000001a363
Oct 16 14:25:03 lnx-16 kernel: FS: 00007fd1e0297700(0000) GS:ffff880c3fc80000(0000) knlGS:0000000000000000
Oct 16 14:25:03 lnx-16 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct 16 14:25:03 lnx-16 kernel: CR2: 00000000041f6050 CR3: 00000003b9998000 CR4: 00000000000007e0
Oct 16 14:25:03 lnx-16 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 16 14:25:03 lnx-16 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 16 14:25:03 lnx-16 kernel: Process ipython (pid: 32537, threadinfo ffff880500318000, task ffff880956814110)
Oct 16 14:25:03 lnx-16 kernel: Stack:
Oct 16 14:25:04 lnx-16 kernel: ffff880500319848 ffffffff8114a715 0000000000000001 0000000000068d8c
Oct 16 14:25:04 lnx-16 kernel: ffff88092537f0c0 ffff88092537f0c0 000000000001a363 000000000001a363
Oct 16 14:25:04 lnx-16 kernel: ffff880500319878 ffffffffa02b753b 00ffffff00000000 ffff881356195480
Oct 16 14:25:04 lnx-16 kernel: Call Trace:
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff8114a715>] ? keyring_search_aux+0x245/0x460
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa02b753b>] afs_GetDCache+0x120b/0x22d0 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff81169883>] ? cpumask_next_and+0x23/0x40
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa02ba5d1>] afs_ObtainDCacheForWriting+0x221/0x470 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff810a51e9>] ? zone_statistics+0x99/0xc0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa028effe>] ? PagInCred+0x1e/0x30 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa02ea80a>] afs_UFSWrite+0x22a/0x610 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa02a5110>] ? afs_pag_describe+0x50/0x50 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa02ff43e>] afs_linux_page_writeback+0x10e/0x220 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa0300298>] afs_linux_write_end+0x138/0x350 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff8108f1b0>] ? grab_cache_page_write_begin+0x90/0x100
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa030130a>] ? afs_linux_write_begin+0x3a/0x180 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff8108e233>] generic_file_buffered_write+0x193/0x2a0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff810e392b>] ? file_update_time+0x9b/0xe0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff810e392b>] ? file_update_time+0x9b/0xe0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff8108fa09>] __generic_file_aio_write+0x1b9/0x3b0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff8131b11f>] ? __mutex_lock_slowpath+0x10f/0x160
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff8108fc7a>] generic_file_aio_write+0x7a/0xf0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff810c9997>] do_sync_write+0xa7/0xe0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffffa02fde62>] afs_linux_write+0x232/0x480 [openafs]
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff81151217>] ? security_file_permission+0x27/0xb0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff810ca1e3>] vfs_write+0xa3/0x160
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff810ca4e5>] sys_write+0x45/0xa0
Oct 16 14:25:04 lnx-16 kernel: [<ffffffff8131dbe2>] system_call_fastpath+0x16/0x1b
Oct 16 14:25:04 lnx-16 kernel: Code: a0 31 c0 e8 49 22 06 e1 0f 0b b9 e8 05 00 00 48 c7 c2 a0 29 31 a0 48 c7 c6 1b 67 31 a0 48 c7 c7 e0 29 31 a0 31 c0 e8 26 22 06 e1 <0f> 0b b9 f1 05 00 00 48 c7 c2 a0 29 31 a0 48 c7 c6 1b 67 31 a0
Oct 16 14:25:04 lnx-16 kernel: RIP [<ffffffffa02b6219>] afs_AllocDCache+0x469/0x4c0 [openafs]
Oct 16 14:25:04 lnx-16 kernel: RSP <ffff880500319708>
Oct 16 14:25:04 lnx-16 kernel: ---[ end trace 4804a936569fd755 ]---

--
Hans-Werner Paulsen hans@MPA-Garching.MPG.DE
MPI für Astrophysik Tel 089-30000-2602
Karl-Schwarzschild-Str. 1 Fax 089-30000-2235
D-85741 Garching

gerrits 10354-10358 should avoid this panic. (If you want to run with those, you'll probably need
other patches from 1.6.x, or you can just run the head of 1.6.x with those.) I believe the current
plan is to get those into 1.6.6, barring any new issues coming up.

I haven't looked much yet into the underlying cause of the i/o errors in the first place, though, so
that's still pending.

--
Andrew Deason
adeason@sinenomine.net