[ovirt-users] VM get stuck randomly

[Christophe TREFOIS]

Hi Yaniv,

See my answers / questions below under [CT].

From: Yaniv Kaul [mailto:ykaul at redhat.com]
Sent: dimanche 13 mars 2016 12:08
To: Christophe TREFOIS <christophe.trefois at uni.lu>
Cc: users <users at ovirt.org>
Subject: Re: [ovirt-users] VM get stuck randomly



On Sun, Mar 13, 2016 at 9:46 AM, Christophe TREFOIS <christophe.trefois at uni.lu<mailto:christophe.trefois at uni.lu>> wrote:
Dear all,

I have a problem since couple of weeks, where randomly 1 VM (not always the same) becomes completely unresponsive.
We find this out because our Icinga server complains that host is down.

Upon inspection, we find we can’t open a console to the VM, nor can we login.

I assume 3.6's console feature, or is it Spice/VNC?
[CT]

This is 3.5, VNC/Spice yes. Sometimes we can connect, but there’s no way to do anything, eg type or so on.


In oVirt engine, the VM looks like “up”. The only weird thing is that RAM usage shows 0% and CPU usage shows 100% or 75% depending on number of cores.

Any chance there's really something bad going on within the VM? Anything in its journal or /var/log/messages or ... depending on the OS?
Y.
[CT]
It is possible. It seems to be mostly VMs with Ubuntu 14.04 and latest kernels. I read somewhere, I couldn’t find now, that there’s perhaps a bug in 3.x kernel with regards to libvirt / vdsm. But my knowledge is too limited to even know where to begin the investigation ☺

On the VM logs, we just see normal VM stuff, then nothing, and then when the VM was rebooted, there’s a couple of lines of ^@^@^@ characters repeating. But nothing else really.
Initially we thought it’s a bug with aufs on Docker, but the machines getting stuck now don’t run either.

From your answer, I deduce that if vdsm or libvirt or spm would see a problem with storage / memory / cpu, it would suspend the VM and provide that info to ovirt-engine?
Since this is not happening, you think it could be related to the inside of the VM rather than the oVirt environment, correct?

Thank you for your help (especially on a Sunday) ☺


The only way to recover is to force shutdown the VM via 2-times shutdown from the engine.

Could you please help me to start debugging this?
I can provide any logs, but I’m not sure which ones, because I couldn’t see anything with ERROR in the vdsm logs on the host.

The host is running

OS Version:             RHEL - 7 - 1.1503.el7.centos.2.8
Kernel Version: 3.10.0 - 229.14.1.el7.x86_64
KVM Version:            2.1.2 - 23.el7_1.8.1
LIBVIRT Version:        libvirt-1.2.8-16.el7_1.4
VDSM Version:   vdsm-4.16.26-0.el7.centos
SPICE Version:  0.12.4 - 9.el7_1.3
GlusterFS Version:      glusterfs-3.7.5-1.el7

We use a locally exported gluster as storage domain (eg, storage is on the same machine exposed via gluster). No replica.
We run around 50 VMs on that host.

Thank you for your help in this,

—
Christophe


_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160313/0f31a60c/attachment-0001.html>