I am running into panics on my ovirt hosts and wanted to know if anyone has seen the same thing. I have 18 ovirt hosts and so far 8 of them have crashed. I have another 18 hosts running KVM on Fedora 16 that are doing fine. There are hardware differences besides oVirt, however they are relatively minor (or so we think). [2299856.345978] systemd[1]: segfault at 7fff36f4df40 ip 000000000040b38c sp 00007fff36f4df30 error 6 in systemd[400000+ce000] [2299856.464068] Kernel panic - not syncing: Attempted to kill init! [2299856.470172] Pid: 1, comm: systemd Tainted: G D 3.2.7-1.fc16.x86_64 #1 [2299856.477656] Call Trace: [2299856.480293] [<ffffffff815d7510>] panic+0x91/0x1a7 [2299856.485265] [<ffffffff81072541>] do_exit+0x861/0x8a0 [2299856.490498] [<ffffffff810728d2>] do_group_exit+0x42/0xa0 [2299856.496081] [<ffffffff810826f6>] get_signal_to_deliver+0x206/0x5b0 [2299856.502526] [<ffffffff81014165>] do_signal+0x65/0x770 [2299856.507847] [<ffffffff815d7677>] ? printk+0x51/0x53 [2299856.512994] [<ffffffff81014918>] do_notify_resume+0x88/0xb0 [2299856.518834] [<ffffffff815e22fc>] retint_signal+0x48/0x8c ---- [88255.648967] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [88255.649918] IP: [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs] [88255.649918] PGD 40090c067 PUD 3c6225067 PMD 0 [88255.649918] Oops: 0000 [#1] SMP [88255.649918] CPU 1 [88255.649918] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 8021q garp serio_raw i2c_i801 microcode i5000_edac iTCO_wdt edac_core iTCO_vendor_support ioatdma i5k_amb dca shpchp vhost_net macvtap macvlan tun virtio_net kvm_intel kvm bridge stp llc bonding e1000e radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_multipath nfs lockd fscache auth_rpcgss nfs_acl sunrpc [last unloaded: scsi_wait_scan] [88255.649918] [88255.649918] Pid: 790, comm: updatedb Not tainted 3.2.7-1.fc16.x86_64 #1 Rackable Systems Inc. S5000PSL/S5000PSL [88255.649918] RIP: 0010:[<ffffffffa008b191>] [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs] [88255.649918] RSP: 0018:ffff8803c6117b48 EFLAGS: 00010286 [88255.649918] RAX: ffffffffa00c5600 RBX: ffff8803ed090e40 RCX: 000000000000001c [88255.649918] RDX: ffff8803c6117c20 RSI: 0000000000000000 RDI: ffff8803ed090e40 [88255.649918] RBP: ffff8803c6117b88 R08: 000000000000001c R09: ffff8803ed090e78 [88255.649918] R10: ffff8803ed090e40 R11: 0000000000000002 R12: ffff880402565300 [88255.649918] R13: ffff8803c6117be8 R14: 0000000000000000 R15: ffff8803ffe8b100 [88255.649918] FS: 00007f1f008a4700(0000) GS:ffff88041fc40000(0000) knlGS:0000000000000000 [88255.649918] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [88255.649918] CR2: 0000000000000038 CR3: 00000003c60b8000 CR4: 00000000000006e0 [88255.649918] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [88255.649918] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [88255.649918] Process updatedb (pid: 790, threadinfo ffff8803c6116000, task ffff8803c62bae40) [88255.649918] Stack: [88255.649918] 0000000000000000 ffff8803ffe8b100 ffff8803c6117b88 ffff8803ed090e40 [88255.649918] ffff880402565300 ffff8803c6117be8 0000000000000000 ffff8803ffe8b100 [88255.649918] ffff8803c6117bc8 ffffffff81184680 ffff8803c6117ba8 ffffffff8126369c [88255.649918] Call Trace: [88255.649918] [<ffffffff81184680>] __lookup_hash.part.8+0x90/0xe0 [88255.649918] [<ffffffff8126369c>] ? security_inode_permission+0x1c/0x30 [88255.649918] [<ffffffff81184b7e>] lookup_one_len+0xee/0x120 [88255.649918] [<ffffffff81184978>] ? inode_permission+0x48/0x100 [88255.649918] [<ffffffffa009bff3>] nfs_sillyrename+0x103/0x520 [nfs] [88255.649918] [<ffffffff81165bbd>] ? kmem_cache_alloc+0x11d/0x140 [88255.649918] [<ffffffff8118ff44>] ? __d_alloc+0x34/0x180 [88255.649918] [<ffffffff81190040>] ? __d_alloc+0x130/0x180 [88255.649918] [<ffffffffa008aecc>] nfs_rename+0x1cc/0x240 [nfs] [88255.649918] [<ffffffff81185d35>] vfs_rename+0x2b5/0x460 [88255.649918] [<ffffffff81184680>] ? __lookup_hash.part.8+0x90/0xe0 [88255.649918] [<ffffffff81189473>] sys_renameat+0x1f3/0x220 [88255.649918] [<ffffffff81193823>] ? notify_change+0x253/0x340 [88255.649918] [<ffffffff81196aa4>] ? mntput_no_expire+0x24/0x100 [88255.649918] [<ffffffff81196b9f>] ? mntput+0x1f/0x30 [88255.649918] [<ffffffff81183c72>] ? path_put+0x22/0x30 [88255.649918] [<ffffffff811894bb>] sys_rename+0x1b/0x20 [88255.649918] [<ffffffff815e9d82>] system_call_fastpath+0x16/0x1b [88255.649918] Code: 54 e1 e9 4c fe ff ff 0f 1f 00 55 48 89 e5 48 83 ec 40 48 89 5d d8 4c 89 65 e0 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 66 66 66 66 90 <f6> 46 38 40 b8 f6 ff ff ff 49 89 fd 49 89 f7 0f 85 d6 00 00 00 [88255.649918] RIP [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs] [88255.649918] RSP <ffff8803c6117b48> [88255.649918] CR2: 0000000000000038 [88256.012265] ---[ end trace 4301002d5bb8ae27 ]--- ---- [2379039.703224] systemd[1]: segfault at 7fffa79e8f18 ip 00000000004661ee sp 00007fffa79e8f10 error 6 in systemd[400000+ce000] [2379039.781077] Kernel panic - not syncing: Attempted to kill init! [2379039.787304] Pid: 1, comm: systemd Tainted: G D 3.2.7-1.fc16.x86_64 #1 [2379039.794886] Call Trace: [2379039.797564] [<ffffffff815d7510>] panic+0x91/0x1a7 [2379039.802618] [<ffffffff81072541>] do_exit+0x861/0x8a0 [2379039.807937] [<ffffffff810728d2>] do_group_exit+0x42/0xa0 [2379039.813623] [<ffffffff810826f6>] get_signal_to_deliver+0x206/0x5b0 [2379039.820161] [<ffffffff81014165>] do_signal+0x65/0x770 [2379039.825563] [<ffffffff815d7677>] ? printk+0x51/0x53 [2379039.830787] [<ffffffff8106d043>] ? do_fork+0x173/0x330 [2379039.836279] [<ffffffff81014918>] do_notify_resume+0x88/0xb0 [2379039.842220] [<ffffffff815e22fc>] retint_signal+0x48/0x8c ---- [88539.434313] BUG: unable to handle kernel NULL pointer dereference at 0000000000000038 [88539.435257] IP: [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs] [88539.435257] PGD 27dbd5067 PUD 2953af067 PMD 0 [88539.435257] Oops: 0000 [#1] SMP [88539.435257] CPU 1 [88539.435257] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 8021q garp iTCO_wdt iTCO_vendor_support i5000_edac edac_core i2c_i801 microcode serio_raw i5k_amb ioatdma dca shpchp vhost_net macvtap macvlan tun virtio_net kvm_intel kvm bridge stp llc bonding e1000e radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_multipath nfs lockd fscache auth_rpcgss nfs_acl sunrpc [last unloaded: scsi_wait_scan] [88539.435257] [88539.435257] Pid: 5947, comm: updatedb Not tainted 3.2.7-1.fc16.x86_64 #1 Rackable Systems Inc. S5000PSL/S5000PSL [88539.435257] RIP: 0010:[<ffffffffa008b191>] [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs] [88539.435257] RSP: 0018:ffff8802490a1b48 EFLAGS: 00010286 [88539.435257] RAX: ffffffffa00c5600 RBX: ffff8802f35f3e40 RCX: 000000000000001c [88539.435257] RDX: ffff8802490a1c20 RSI: 0000000000000000 RDI: ffff8802f35f3e40 [88539.435257] RBP: ffff8802490a1b88 R08: 000000000000001c R09: ffff8802f35f3e78 [88539.435257] R10: ffff8802f35f3e40 R11: 0000000000000001 R12: ffff8803066373c0 [88539.435257] R13: ffff8802490a1be8 R14: 0000000000000000 R15: ffff880307a9b8e0 [88539.435257] FS: 00007fdd293bc700(0000) GS:ffff88031fc40000(0000) knlGS:0000000000000000 [88539.435257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [88539.435257] CR2: 0000000000000038 CR3: 00000002a5e46000 CR4: 00000000000026e0 [88539.435257] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [88539.435257] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [88539.435257] Process updatedb (pid: 5947, threadinfo ffff8802490a0000, task ffff880300314560) [88539.435257] Stack: [88539.435257] 0000000000000000 ffff880307a9b8e0 ffff8802490a1b88 ffff8802f35f3e40 [88539.435257] ffff8803066373c0 ffff8802490a1be8 0000000000000000 ffff880307a9b8e0 [88539.435257] ffff8802490a1bc8 ffffffff81184680 ffff8802490a1ba8 ffffffff8126369c [88539.435257] Call Trace: [88539.435257] [<ffffffff81184680>] __lookup_hash.part.8+0x90/0xe0 [88539.435257] [<ffffffff8126369c>] ? security_inode_permission+0x1c/0x30 [88539.435257] [<ffffffff81184b7e>] lookup_one_len+0xee/0x120 [88539.435257] [<ffffffff81184978>] ? inode_permission+0x48/0x100 [88539.435257] [<ffffffffa009bff3>] nfs_sillyrename+0x103/0x520 [nfs] [88539.435257] [<ffffffff81165bbd>] ? kmem_cache_alloc+0x11d/0x140 [88539.435257] [<ffffffff8118ff44>] ? __d_alloc+0x34/0x180 [88539.435257] [<ffffffff81190040>] ? __d_alloc+0x130/0x180 [88539.435257] [<ffffffffa008aecc>] nfs_rename+0x1cc/0x240 [nfs] [88539.435257] [<ffffffff81185d35>] vfs_rename+0x2b5/0x460 [88539.435257] [<ffffffff81184680>] ? __lookup_hash.part.8+0x90/0xe0 [88539.435257] [<ffffffff81189473>] sys_renameat+0x1f3/0x220 [88539.435257] [<ffffffff81193823>] ? notify_change+0x253/0x340 [88539.435257] [<ffffffff81196aa4>] ? mntput_no_expire+0x24/0x100 [88539.435257] [<ffffffff81196b9f>] ? mntput+0x1f/0x30 [88539.435257] [<ffffffff81183c72>] ? path_put+0x22/0x30 [88539.435257] [<ffffffff811894bb>] sys_rename+0x1b/0x20 [88539.435257] [<ffffffff815e9d82>] system_call_fastpath+0x16/0x1b [88539.435257] Code: 54 e1 e9 4c fe ff ff 0f 1f 00 55 48 89 e5 48 83 ec 40 48 89 5d d8 4c 89 65 e0 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 66 66 66 66 90 <f6> 46 38 40 b8 f6 ff ff ff 49 89 fd 49 89 f7 0f 85 d6 00 00 00 [88539.435257] RIP [<ffffffffa008b191>] nfs_lookup_revalidate+0x21/0x490 [nfs] [88539.435257] RSP <ffff8802490a1b48> [88539.435257] CR2: 0000000000000038 [88539.797861] ---[ end trace a1ed36439ad7c011 ]---

<> Nathan Stratton nathan at robotics.net http://www.robotics.net