8 p.m.
I am running into panics on my ovirt hosts and wanted to know if anyone
has seen the same thing. I have 18 ovirt hosts and so far 8 of them have
crashed. I have another 18 hosts running KVM on Fedora 16 that are doing
fine. There are hardware differences besides oVirt, however they are
relatively minor (or so we think).
[2299856.345978] systemd[1]: segfault at 7fff36f4df40 ip 000000000040b38c sp
00007fff36f4df30 error 6 in systemd[400000+ce000]
[2299856.464068] Kernel panic - not syncing: Attempted to kill init!
[2299856.470172] Pid: 1, comm: systemd Tainted: G D 3.2.7-1.fc16.x86_64 #1
[2299856.477656] Call Trace:
[2299856.480293] [<ffffffff815d7510>] panic+0x91/0x1a7
[2299856.485265] [<ffffffff81072541>] do_exit+0x861/0x8a0
[2299856.490498] [<ffffffff810728d2>] do_group_exit+0x42/0xa0
[2299856.496081] [<ffffffff810826f6>] get_signal_to_deliver+0x206/0x5b0
[2299856.502526] [<ffffffff81014165>] do_signal+0x65/0x770
[2299856.507847] [<ffffffff815d7677>] ? printk+0x51/0x53
[2299856.512994] [<ffffffff81014918>] do_notify_resume+0x88/0xb0
[2299856.518834] [<ffffffff815e22fc>] retint_signal+0x48/0x8c
----
[88255.648967] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
[88255.649918] IP: [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs]
[88255.649918] PGD 40090c067 PUD 3c6225067 PMD 0
[88255.649918] Oops: 0000 [#1] SMP
[88255.649918] CPU 1
[88255.649918] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm
ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi 8021q garp serio_raw i2c_i801 microcode i5000_edac iTCO_wdt edac_core
iTCO_vendor_support ioatdma i5k_amb dca shpchp vhost_net macvtap macvlan tun virtio_net
kvm_intel kvm bridge stp llc bonding e1000e radeon ttm drm_kms_helper drm i2c_algo_bit
i2c_core dm_multipath nfs lockd fscache auth_rpcgss nfs_acl sunrpc [last unloaded:
scsi_wait_scan]
[88255.649918]
[88255.649918] Pid: 790, comm: updatedb Not tainted 3.2.7-1.fc16.x86_64 #1 Rackable
Systems Inc. S5000PSL/S5000PSL
[88255.649918] RIP: 0010:[<ffffffffa008b191>] [<ffffffffa008b191>]
nfs_lookup_revalidate+0x21/0x490 [nfs]
[88255.649918] RSP: 0018:ffff8803c6117b48 EFLAGS: 00010286
[88255.649918] RAX: ffffffffa00c5600 RBX: ffff8803ed090e40 RCX: 000000000000001c
[88255.649918] RDX: ffff8803c6117c20 RSI: 0000000000000000 RDI: ffff8803ed090e40
[88255.649918] RBP: ffff8803c6117b88 R08: 000000000000001c R09: ffff8803ed090e78
[88255.649918] R10: ffff8803ed090e40 R11: 0000000000000002 R12: ffff880402565300
[88255.649918] R13: ffff8803c6117be8 R14: 0000000000000000 R15: ffff8803ffe8b100
[88255.649918] FS: 00007f1f008a4700(0000) GS:ffff88041fc40000(0000)
knlGS:0000000000000000
[88255.649918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[88255.649918] CR2: 0000000000000038 CR3: 00000003c60b8000 CR4: 00000000000006e0
[88255.649918] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[88255.649918] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[88255.649918] Process updatedb (pid: 790, threadinfo ffff8803c6116000, task
ffff8803c62bae40)
[88255.649918] Stack:
[88255.649918] 0000000000000000 ffff8803ffe8b100 ffff8803c6117b88 ffff8803ed090e40
[88255.649918] ffff880402565300 ffff8803c6117be8 0000000000000000 ffff8803ffe8b100
[88255.649918] ffff8803c6117bc8 ffffffff81184680 ffff8803c6117ba8 ffffffff8126369c
[88255.649918] Call Trace:
[88255.649918] [<ffffffff81184680>] __lookup_hash.part.8+0x90/0xe0
[88255.649918] [<ffffffff8126369c>] ? security_inode_permission+0x1c/0x30
[88255.649918] [<ffffffff81184b7e>] lookup_one_len+0xee/0x120
[88255.649918] [<ffffffff81184978>] ? inode_permission+0x48/0x100
[88255.649918] [<ffffffffa009bff3>] nfs_sillyrename+0x103/0x520 [nfs]
[88255.649918] [<ffffffff81165bbd>] ? kmem_cache_alloc+0x11d/0x140
[88255.649918] [<ffffffff8118ff44>] ? __d_alloc+0x34/0x180
[88255.649918] [<ffffffff81190040>] ? __d_alloc+0x130/0x180
[88255.649918] [<ffffffffa008aecc>] nfs_rename+0x1cc/0x240 [nfs]
[88255.649918] [<ffffffff81185d35>] vfs_rename+0x2b5/0x460
[88255.649918] [<ffffffff81184680>] ? __lookup_hash.part.8+0x90/0xe0
[88255.649918] [<ffffffff81189473>] sys_renameat+0x1f3/0x220
[88255.649918] [<ffffffff81193823>] ? notify_change+0x253/0x340
[88255.649918] [<ffffffff81196aa4>] ? mntput_no_expire+0x24/0x100
[88255.649918] [<ffffffff81196b9f>] ? mntput+0x1f/0x30
[88255.649918] [<ffffffff81183c72>] ? path_put+0x22/0x30
[88255.649918] [<ffffffff811894bb>] sys_rename+0x1b/0x20
[88255.649918] [<ffffffff815e9d82>] system_call_fastpath+0x16/0x1b
[88255.649918] Code: 54 e1 e9 4c fe ff ff 0f 1f 00 55 48 89 e5 48 83 ec 40 48 89 5d d8 4c
89 65 e0 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 66 66 66 66 90 <f6> 46 38 40 b8 f6 ff
ff ff 49 89 fd 49 89 f7 0f 85 d6 00 00 00
[88255.649918] RIP [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs]
[88255.649918] RSP <ffff8803c6117b48>
[88255.649918] CR2: 0000000000000038
[88256.012265] ---[ end trace 4301002d5bb8ae27 ]---
----
[2379039.703224] systemd[1]: segfault at 7fffa79e8f18 ip 00000000004661ee sp
00007fffa79e8f10 error 6 in systemd[400000+ce000]
[2379039.781077] Kernel panic - not syncing: Attempted to kill init!
[2379039.787304] Pid: 1, comm: systemd Tainted: G D 3.2.7-1.fc16.x86_64 #1
[2379039.794886] Call Trace:
[2379039.797564] [<ffffffff815d7510>] panic+0x91/0x1a7
[2379039.802618] [<ffffffff81072541>] do_exit+0x861/0x8a0
[2379039.807937] [<ffffffff810728d2>] do_group_exit+0x42/0xa0
[2379039.813623] [<ffffffff810826f6>] get_signal_to_deliver+0x206/0x5b0
[2379039.820161] [<ffffffff81014165>] do_signal+0x65/0x770
[2379039.825563] [<ffffffff815d7677>] ? printk+0x51/0x53
[2379039.830787] [<ffffffff8106d043>] ? do_fork+0x173/0x330
[2379039.836279] [<ffffffff81014918>] do_notify_resume+0x88/0xb0
[2379039.842220] [<ffffffff815e22fc>] retint_signal+0x48/0x8c
----
[88539.434313] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038
[88539.435257] IP: [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs]
[88539.435257] PGD 27dbd5067 PUD 2953af067 PMD 0
[88539.435257] Oops: 0000 [#1] SMP
[88539.435257] CPU 1
[88539.435257] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables be2iscsi
iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm
ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi 8021q garp iTCO_wdt iTCO_vendor_support i5000_edac edac_core i2c_i801
microcode serio_raw i5k_amb ioatdma dca shpchp vhost_net macvtap macvlan tun virtio_net
kvm_intel kvm bridge stp llc bonding e1000e radeon ttm drm_kms_helper drm i2c_algo_bit
i2c_core dm_multipath nfs lockd fscache auth_rpcgss nfs_acl sunrpc [last unloaded:
scsi_wait_scan]
[88539.435257]
[88539.435257] Pid: 5947, comm: updatedb Not tainted 3.2.7-1.fc16.x86_64 #1 Rackable
Systems Inc. S5000PSL/S5000PSL
[88539.435257] RIP: 0010:[<ffffffffa008b191>] [<ffffffffa008b191>]
nfs_lookup_revalidate+0x21/0x490 [nfs]
[88539.435257] RSP: 0018:ffff8802490a1b48 EFLAGS: 00010286
[88539.435257] RAX: ffffffffa00c5600 RBX: ffff8802f35f3e40 RCX: 000000000000001c
[88539.435257] RDX: ffff8802490a1c20 RSI: 0000000000000000 RDI: ffff8802f35f3e40
[88539.435257] RBP: ffff8802490a1b88 R08: 000000000000001c R09: ffff8802f35f3e78
[88539.435257] R10: ffff8802f35f3e40 R11: 0000000000000001 R12: ffff8803066373c0
[88539.435257] R13: ffff8802490a1be8 R14: 0000000000000000 R15: ffff880307a9b8e0
[88539.435257] FS: 00007fdd293bc700(0000) GS:ffff88031fc40000(0000)
knlGS:0000000000000000
[88539.435257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[88539.435257] CR2: 0000000000000038 CR3: 00000002a5e46000 CR4: 00000000000026e0
[88539.435257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[88539.435257] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[88539.435257] Process updatedb (pid: 5947, threadinfo ffff8802490a0000, task
ffff880300314560)
[88539.435257] Stack:
[88539.435257] 0000000000000000 ffff880307a9b8e0 ffff8802490a1b88 ffff8802f35f3e40
[88539.435257] ffff8803066373c0 ffff8802490a1be8 0000000000000000 ffff880307a9b8e0
[88539.435257] ffff8802490a1bc8 ffffffff81184680 ffff8802490a1ba8 ffffffff8126369c
[88539.435257] Call Trace:
[88539.435257] [<ffffffff81184680>] __lookup_hash.part.8+0x90/0xe0
[88539.435257] [<ffffffff8126369c>] ? security_inode_permission+0x1c/0x30
[88539.435257] [<ffffffff81184b7e>] lookup_one_len+0xee/0x120
[88539.435257] [<ffffffff81184978>] ? inode_permission+0x48/0x100
[88539.435257] [<ffffffffa009bff3>] nfs_sillyrename+0x103/0x520 [nfs]
[88539.435257] [<ffffffff81165bbd>] ? kmem_cache_alloc+0x11d/0x140
[88539.435257] [<ffffffff8118ff44>] ? __d_alloc+0x34/0x180
[88539.435257] [<ffffffff81190040>] ? __d_alloc+0x130/0x180
[88539.435257] [<ffffffffa008aecc>] nfs_rename+0x1cc/0x240 [nfs]
[88539.435257] [<ffffffff81185d35>] vfs_rename+0x2b5/0x460
[88539.435257] [<ffffffff81184680>] ? __lookup_hash.part.8+0x90/0xe0
[88539.435257] [<ffffffff81189473>] sys_renameat+0x1f3/0x220
[88539.435257] [<ffffffff81193823>] ? notify_change+0x253/0x340
[88539.435257] [<ffffffff81196aa4>] ? mntput_no_expire+0x24/0x100
[88539.435257] [<ffffffff81196b9f>] ? mntput+0x1f/0x30
[88539.435257] [<ffffffff81183c72>] ? path_put+0x22/0x30
[88539.435257] [<ffffffff811894bb>] sys_rename+0x1b/0x20
[88539.435257] [<ffffffff815e9d82>] system_call_fastpath+0x16/0x1b
[88539.435257] Code: 54 e1 e9 4c fe ff ff 0f 1f 00 55 48 89 e5 48 83 ec 40 48 89 5d d8 4c
89 65 e0 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 66 66 66 66 90 <f6> 46 38 40 b8 f6 ff
ff ff 49 89 fd 49 89 f7 0f 85 d6 00 00 00
[88539.435257] RIP [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs]
[88539.435257] RSP <ffff8802490a1b48>
[88539.435257] CR2: 0000000000000038
[88539.797861] ---[ end trace a1ed36439ad7c011 ]---
<>
Nathan Stratton
nathan at robotics.net
http://www.robotics.net