[Users] Starting VM gets paused

Fri Jun 14 11:32:49 UTC 2013

Nicolas Ecarnot <nicolas at ...> writes:

> 
> Le 20/04/2013 22:55, Itamar Heim a écrit :
> > On 03/27/2013 10:38 AM, Nicolas Ecarnot wrote:
> >> Le 26/03/2013 12:17, Nicolas Ecarnot a écrit :
> >>> Le 25/03/2013 12:10, Nicolas Ecarnot a écrit :
> >>>> Le 24/03/2013 09:53, Dafna Ron a écrit :
> >>>>> is the vm preallocated or thin provision disk type?
> >>>>
> >>>> This VM has 3 disks :
> >>>> - first disk to host the windows system : Thin provision
> >>>> - second disk to store some data : Preallocated
> >>>> - third disk to store some more data : Thin provision
> >>>>
> >>>> I'm realizing that amongst the 15 VMs, only this one and another one
> >>>> that is stopped are using preallocated disks.
> >>>> I'm regularly migrating some VMs (and stopping and starting and 
playing
> >>>> with them) with no issue, and they all are using thin provisioned
> >>>> disks!
> >>>>
> >>>> Could this be a common factor of the problem?
> >>>>
> >>>>>
> >>>>> also, can you please attach engine, vdsm, libvirt and the vm's qemu
> >>>>> logs?
> >>>>
> >>>> Relevant logs :
> >>>>
> >>>> ############
> >>>>
> >>>> Ok, I'm in the process of collecting the logs and posting them in a
> >>>> useable manner.
> >>>>
> >>>> More to come.
> >>>
> >>> Ok, once again, I ran a test and observed the relevant logs.
> >>> I tried to isolate the time frames, but it may be long for vdsm.log
> >>>
> >>> Here they are :
> >>> * /var/log/libvirt/qemu/serv-chk-adm3.log
> >>> http://pastebin.com/JVKMSmxD
> >>> * /var/log/libvirtd.log
> >>> http://pastebin.com/sWGDCqNh
> >>> * /var/log/vdsm/vdsm.log (the BIG one)
> >>> http://pastebin.com/bevTEhym
> >>>
> >>> What I can add to help you help me, is that :
> >>> - I saw that all my VM appear as tainted. I did not know what that 
meant
> >>> (but RTFMed since), and this does not appear to disturb the other VMs
> >>> - Many VMs including the problematic one have been imported from
> >>> ovirt-v2v with now such issue.
> >>> - This particular VM was also imported, but the starting point was a
> >>> vmdk or ova single file.
> >>> - Two additionnal data disks were added
> >>> - As I said, this is the only running VM stored as pre allocated.
> >>>
> >>> Regards,
> >>>
> >>
> >> One suggestion : I see no obvious errors in the log files. Could this
> >> paused state happen due to a VM's kernel panic?
> >>
> >
> > is this still relevant?
> 
> It is!
> Further investigations from my colleague shown the following facts :
> - This VM has 3 disks. Only one of those disks is responsible for the 
> problem
> - In this disk, my coworker has found only 3 files (database files) that 
> he can do nothing with without leading to the freeze.
> - He tried to cat them into /dev/null, and this is leading to the freeze
> - He tried to copy them into another disk -> freeze!
> 
> We see absolutely no evidence of a kernel panic.
> Rather, this seems to be related to a network bottleneck between the 
> node and the iSCSI SAN, leading to oVirt unable to sustain a sufficent 
> bandwidth and freezing the VM.
> 
> Since then, we moved to another solution, but for the sake of opensource 
> debugging, we did kept the faulty VM for your eyes only :)
> 

Hi, anyone has answer? I got same issue. I create new vm and cannot start on 
ovirt node. The status wating for launch long time and vm cannot start.
here is vdsm log:
VM Channels Listener::DEBUG::2013-06-14 18:32:32,504::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 37.
VM Channels Listener::DEBUG::2013-06-14 18:32:32,504::guestIF::95::vm.Vm::
(_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting 
connection to /var/lib/libvirt/qemu/channels/4000570-
01.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 32.
VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::guestIF::95::vm.Vm::
(_connect) vmId=`187f61c9-d81f-491a-b5f0-4798ec6c8342`::Attempting 
connection to /var/lib/libvirt/qemu/channels/4000565-
01.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 33.
VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::guestIF::95::vm.Vm::
(_connect) vmId=`6c3074ae-c752-4622-94e7-a4ca09b252f7`::Attempting 
connection to /var/lib/libvirt/qemu/channels/4000563-
02.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 35.
VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::guestIF::95::vm.Vm::
(_connect) vmId=`20f144cd-f027-4710-a433-dcdc62eec554`::Attempting 
connection to /var/lib/libvirt/qemu/channels/4000568-
01.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 37.
VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::guestIF::95::vm.Vm::
(_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting 
connection to /var/lib/libvirt/qemu/channels/4000570-
01.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2013-06-14 18:32:34,507::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 32.
VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::guestIF::95::vm.Vm::
(_connect) vmId=`187f61c9-d81f-491a-b5f0-4798ec6c8342`::Attempting 
connection to /var/lib/libvirt/qemu/channels/4000565-
01.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 33.
VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::guestIF::95::vm.Vm::
(_connect) vmId=`6c3074ae-c752-4622-94e7-a4ca09b252f7`::Attempting 
connection to /var/lib/libvirt/qemu/channels/4000563-
02.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 35.
VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::guestIF::95::vm.Vm::
(_connect) vmId=`20f144cd-f027-4710-a433-dcdc62eec554`::Attempting 
connection to /var/lib/libvirt/qemu/channels/4000568-
01.com.redhat.rhevm.vdsm
VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::vmChannels::104::vds::
(_handle_unconnected) Trying to connect fileno 37.
VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::guestIF::95::vm.Vm::
(_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting 
connection to /var/lib/libvirt/q