vdsmd and libvirtd services failed to start
Hi, I have a HCI setup running on 3 nodes and created 6 VM's. Was running IO (like dd and linux untar) on those VM's overnight. Next day i saw that for 2 of the nodes vdsmd and libvirtd services failed and if manually started, they don't come up. All the VM's state has changed to 'unknown' and failed to migrate. Can someone help looking at the logs and figure out the RCA. Let me know what all logs are needed, i can post the same. Thanks, Bhaskarakiran.
On 05 May 2016, at 16:18, Bhaskarakiran <byarlaga@redhat.com> wrote: =20 Hi, =20 I have a HCI setup running on 3 nodes and created 6 VM's. Was running = IO (like dd and linux untar) on those VM's overnight. Next day i saw =
--Apple-Mail=_70A353A7-A0ED-4E2E-A304-9C0BD71D7884 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 that for 2 of the nodes vdsmd and libvirtd services failed and if = manually started, they don't come up. All the VM's state has changed to = 'unknown' and failed to migrate. Can someone help looking at the logs = and figure out the RCA. Let me know what all logs are needed, i can post = the same. Hi, I saw your other bugs you reported already. In all cases it doesn=E2=80=99= t seem like ovirt=E2=80=99s fault (except the buggy vdsm recovery flow), = the underlying reason is that something got broken in either libvirt or = qemu. For that you better enable libvirt debug logging as the default level is = not so useful. You can find more details about logging at = http://www.ovirt.org/develop/developer-guide/vdsm/log-files/ Once you have that please share/send vdsm.log (it=E2=80=99s rotated = often, so check the times to cover the time from VM creation all the way = to failure), libvirt.log with debug info and VM=E2=80=99s qemu log from = /var/log/libvirt/qemu/<vm name>.log Thanks, michal
=20 Thanks, Bhaskarakiran. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
</div><div><br class=3D""></div><div>Once you have that please = share/send vdsm.log (it=E2=80=99s rotated often, so check the times to = cover the time from VM creation all the way to failure), libvirt.log = with debug info and VM=E2=80=99s qemu log from = /var/log/libvirt/qemu/<vm name>.log</div><div><br = class=3D""></div><div>Thanks,</div><div>michal</div><div><br = class=3D""><blockquote type=3D"cite" class=3D""><div class=3D""><div =
--Apple-Mail=_70A353A7-A0ED-4E2E-A304-9C0BD71D7884 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 05 May 2016, at 16:18, Bhaskarakiran <<a = href=3D"mailto:byarlaga@redhat.com" class=3D"">byarlaga@redhat.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"gmail_default" = style=3D"font-family:trebuchet ms,sans-serif">Hi,<br class=3D""><br = class=3D""></div><div class=3D"gmail_default" = style=3D"font-family:trebuchet ms,sans-serif">I have a HCI setup running = on 3 nodes and created 6 VM's. Was running IO (like dd and linux untar) = on those VM's overnight. Next day i saw that for 2 of the nodes vdsmd = and libvirtd services failed and if manually started, they don't come = up. All the VM's state has changed to 'unknown' and failed to migrate. = Can someone help looking at the logs and figure out the RCA. Let me know = what all logs are needed, i can post the same.<br = class=3D""></div></div></div></blockquote><div><br = class=3D""></div>Hi,</div><div>I saw your other bugs you reported = already. In all cases it doesn=E2=80=99t seem like ovirt=E2=80=99s fault = (except the buggy vdsm recovery flow), the underlying reason is that = something got broken in either libvirt or qemu.</div><div>For that you = better enable libvirt debug logging as the default level is not so = useful.</div><div>You can find more details about logging at <a = href=3D"http://www.ovirt.org/develop/developer-guide/vdsm/log-files/" = class=3D"">http://www.ovirt.org/develop/developer-guide/vdsm/log-files/</a= dir=3D"ltr" class=3D""><div class=3D"gmail_default" = style=3D"font-family:trebuchet ms,sans-serif"><br class=3D""></div><div = class=3D"gmail_default" style=3D"font-family:trebuchet = ms,sans-serif">Thanks,<br class=3D""></div><div class=3D"gmail_default" = style=3D"font-family:trebuchet ms,sans-serif">Bhaskarakiran.<br = class=3D""></div></div> _______________________________________________<br class=3D"">Users = mailing list<br class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D"">Users@ovirt.org</a><br = class=3D"">http://lists.ovirt.org/mailman/listinfo/users<br = class=3D""></div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_70A353A7-A0ED-4E2E-A304-9C0BD71D7884--
Okay. I will check this and get back. Thanks, Bhaskarakiran. On Fri, May 6, 2016 at 12:28 PM, Michal Skrivanek < michal.skrivanek@redhat.com> wrote:
On 05 May 2016, at 16:18, Bhaskarakiran <byarlaga@redhat.com> wrote:
Hi,
I have a HCI setup running on 3 nodes and created 6 VM's. Was running IO (like dd and linux untar) on those VM's overnight. Next day i saw that for 2 of the nodes vdsmd and libvirtd services failed and if manually started, they don't come up. All the VM's state has changed to 'unknown' and failed to migrate. Can someone help looking at the logs and figure out the RCA. Let me know what all logs are needed, i can post the same.
Hi, I saw your other bugs you reported already. In all cases it doesn’t seem like ovirt’s fault (except the buggy vdsm recovery flow), the underlying reason is that something got broken in either libvirt or qemu. For that you better enable libvirt debug logging as the default level is not so useful. You can find more details about logging at http://www.ovirt.org/develop/developer-guide/vdsm/log-files/
Once you have that please share/send vdsm.log (it’s rotated often, so check the times to cover the time from VM creation all the way to failure), libvirt.log with debug info and VM’s qemu log from /var/log/libvirt/qemu/<vm name>.log
Thanks, michal
Thanks, Bhaskarakiran. _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Bhaskarakiran -
Michal Skrivanek