[Users] Starting VM gets paused
Hi, After two months of stable usage of this 3.1 oVirt setup, here comes the first blocking issue for which I've no other mean to ask some hint. When I'm starting a VM, the start process is running fine. Being fast enough, we can ssh-connect to it, but 5 seconds later, the VM is paused. In the manager, I see that : 2013-03-22 09:42:57,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder] (QuartzScheduler_Worker-40) Error in parsing vm pause status. Setting value to NONE 2013-03-22 09:42:57,436 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-40) VM serv-chk-adm3 3e17586d-bf8f-465b-8075-defaac90bc95 moved from PoweringUp --> Paused And on the host, I see one warning message, no error msg, and many looping repeated messages : * Warning : Thread-1968::WARNING::2013-03-22 09:19:18,536::libvirtvm::1547::vm.Vm::(_readPauseCode) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::_readPauseCode unsupported by libvirt vm * Repeated msgs, amongst other repeated ones : Thread-1973::DEBUG::2013-03-22 09:19:20,247::libvirtvm::220::vm.Vm::(_getNetworkStats) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::Network stats not available Thread-1973::DEBUG::2013-03-22 09:19:20,247::libvirtvm::240::vm.Vm::(_getDiskStats) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::Disk hdc stats not available I made my homework and found some bugs that could be similar : https://bugzilla.redhat.com/show_bug.cgi?id=660598 https://bugzilla.redhat.com/show_bug.cgi?id=672208 and moreover : https://bugzilla.redhat.com/show_bug.cgi?id=695393 - I tried to restart the node's vds daemon : same behavior - I tried to reboot the node : same behavior - I tried to to restart the manager's engine : same behavior - I tried to run this VM on another node : same behavior - I tried to run another VM on the node I saw the issue : the other VM is running fine. I don't know if I have to conclude that this issue is specific to this VM, but I sounds like yes. Things to say about this VM : - it it a RH6 IIRC. It has already been successfully started, migrated, stopped and rebooted many times in the past. - it has 3 disks : one for the system and two for datas. - it has no snapshots - it has no different or complicated network setup My storage domain is a SAN, iSCSI linked, and doing good job since months. I must admit I'm a bit stuck. Last thing I haven't tried is to reboot the manager, though I'm not sure that would help. -- Nicolas Ecarnot
I got hit up with the same issue - I think it had something to do with latency or timeout to my NAS/SAN, despite an isolated & prioritised storage network. I'm guessing the VDSM puts VM's in a paused state whenever it hits a slight panic or speed-wobble. I've, for the time-being, moved to a NFS/POSIX share & the issues has not presented again. - J On 22/03/13 22:59, Nicolas Ecarnot wrote:
Hi,
After two months of stable usage of this 3.1 oVirt setup, here comes the first blocking issue for which I've no other mean to ask some hint.
When I'm starting a VM, the start process is running fine. Being fast enough, we can ssh-connect to it, but 5 seconds later, the VM is paused.
In the manager, I see that :
2013-03-22 09:42:57,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder] (QuartzScheduler_Worker-40) Error in parsing vm pause status. Setting value to NONE 2013-03-22 09:42:57,436 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-40) VM serv-chk-adm3 3e17586d-bf8f-465b-8075-defaac90bc95 moved from PoweringUp --> Paused
And on the host, I see one warning message, no error msg, and many looping repeated messages :
* Warning :
Thread-1968::WARNING::2013-03-22 09:19:18,536::libvirtvm::1547::vm.Vm::(_readPauseCode) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::_readPauseCode unsupported by libvirt vm
* Repeated msgs, amongst other repeated ones : Thread-1973::DEBUG::2013-03-22 09:19:20,247::libvirtvm::220::vm.Vm::(_getNetworkStats) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::Network stats not available Thread-1973::DEBUG::2013-03-22 09:19:20,247::libvirtvm::240::vm.Vm::(_getDiskStats) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::Disk hdc stats not available
I made my homework and found some bugs that could be similar : https://bugzilla.redhat.com/show_bug.cgi?id=660598 https://bugzilla.redhat.com/show_bug.cgi?id=672208 and moreover : https://bugzilla.redhat.com/show_bug.cgi?id=695393
- I tried to restart the node's vds daemon : same behavior - I tried to reboot the node : same behavior - I tried to to restart the manager's engine : same behavior - I tried to run this VM on another node : same behavior - I tried to run another VM on the node I saw the issue : the other VM is running fine.
I don't know if I have to conclude that this issue is specific to this VM, but I sounds like yes. Things to say about this VM : - it it a RH6 IIRC. It has already been successfully started, migrated, stopped and rebooted many times in the past. - it has 3 disks : one for the system and two for datas. - it has no snapshots - it has no different or complicated network setup
My storage domain is a SAN, iSCSI linked, and doing good job since months.
I must admit I'm a bit stuck. Last thing I haven't tried is to reboot the manager, though I'm not sure that would help.
Le 22/03/2013 11:29, Jaco a écrit :
I got hit up with the same issue - I think it had something to do with latency or timeout to my NAS/SAN, despite an isolated & prioritised storage network. I'm guessing the VDSM puts VM's in a paused state whenever it hits a slight panic or speed-wobble.
OK, I could understand such a behavior, but creating such an issue on EVERY VM. The case I'm witnessing is that only one VM is hit by this problem. I have to add I also have a completely dedicated network only for iSCSI. -- Nicolas Ecarnot
is the vm preallocated or thin provision disk type? also, can you please attach engine, vdsm, libvirt and the vm's qemu logs? On 03/22/2013 11:59 AM, Nicolas Ecarnot wrote:
Hi,
After two months of stable usage of this 3.1 oVirt setup, here comes the first blocking issue for which I've no other mean to ask some hint.
When I'm starting a VM, the start process is running fine. Being fast enough, we can ssh-connect to it, but 5 seconds later, the VM is paused.
In the manager, I see that :
2013-03-22 09:42:57,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder] (QuartzScheduler_Worker-40) Error in parsing vm pause status. Setting value to NONE 2013-03-22 09:42:57,436 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-40) VM serv-chk-adm3 3e17586d-bf8f-465b-8075-defaac90bc95 moved from PoweringUp --> Paused
And on the host, I see one warning message, no error msg, and many looping repeated messages :
* Warning :
Thread-1968::WARNING::2013-03-22 09:19:18,536::libvirtvm::1547::vm.Vm::(_readPauseCode) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::_readPauseCode unsupported by libvirt vm
* Repeated msgs, amongst other repeated ones : Thread-1973::DEBUG::2013-03-22 09:19:20,247::libvirtvm::220::vm.Vm::(_getNetworkStats) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::Network stats not available Thread-1973::DEBUG::2013-03-22 09:19:20,247::libvirtvm::240::vm.Vm::(_getDiskStats) vmId=`3e17586d-bf8f-465b-8075-defaac90bc95`::Disk hdc stats not available
I made my homework and found some bugs that could be similar : https://bugzilla.redhat.com/show_bug.cgi?id=660598 https://bugzilla.redhat.com/show_bug.cgi?id=672208 and moreover : https://bugzilla.redhat.com/show_bug.cgi?id=695393
- I tried to restart the node's vds daemon : same behavior - I tried to reboot the node : same behavior - I tried to to restart the manager's engine : same behavior - I tried to run this VM on another node : same behavior - I tried to run another VM on the node I saw the issue : the other VM is running fine.
I don't know if I have to conclude that this issue is specific to this VM, but I sounds like yes. Things to say about this VM : - it it a RH6 IIRC. It has already been successfully started, migrated, stopped and rebooted many times in the past. - it has 3 disks : one for the system and two for datas. - it has no snapshots - it has no different or complicated network setup
My storage domain is a SAN, iSCSI linked, and doing good job since months.
I must admit I'm a bit stuck. Last thing I haven't tried is to reboot the manager, though I'm not sure that would help.
-- Dafna Ron
Le 24/03/2013 09:53, Dafna Ron a écrit :
is the vm preallocated or thin provision disk type?
This VM has 3 disks : - first disk to host the windows system : Thin provision - second disk to store some data : Preallocated - third disk to store some more data : Thin provision I'm realizing that amongst the 15 VMs, only this one and another one that is stopped are using preallocated disks. I'm regularly migrating some VMs (and stopping and starting and playing with them) with no issue, and they all are using thin provisioned disks! Could this be a common factor of the problem?
also, can you please attach engine, vdsm, libvirt and the vm's qemu logs?
Relevant logs : ############ Ok, I'm in the process of collecting the logs and posting them in a useable manner. More to come. -- Nicolas Ecarnot
Le 25/03/2013 12:10, Nicolas Ecarnot a écrit :
Le 24/03/2013 09:53, Dafna Ron a écrit :
is the vm preallocated or thin provision disk type?
This VM has 3 disks : - first disk to host the windows system : Thin provision - second disk to store some data : Preallocated - third disk to store some more data : Thin provision
I'm realizing that amongst the 15 VMs, only this one and another one that is stopped are using preallocated disks. I'm regularly migrating some VMs (and stopping and starting and playing with them) with no issue, and they all are using thin provisioned disks!
Could this be a common factor of the problem?
also, can you please attach engine, vdsm, libvirt and the vm's qemu logs?
Relevant logs :
############
Ok, I'm in the process of collecting the logs and posting them in a useable manner.
More to come.
Ok, once again, I ran a test and observed the relevant logs. I tried to isolate the time frames, but it may be long for vdsm.log Here they are : * /var/log/libvirt/qemu/serv-chk-adm3.log http://pastebin.com/JVKMSmxD * /var/log/libvirtd.log http://pastebin.com/sWGDCqNh * /var/log/vdsm/vdsm.log (the BIG one) http://pastebin.com/bevTEhym What I can add to help you help me, is that : - I saw that all my VM appear as tainted. I did not know what that meant (but RTFMed since), and this does not appear to disturb the other VMs - Many VMs including the problematic one have been imported from ovirt-v2v with now such issue. - This particular VM was also imported, but the starting point was a vmdk or ova single file. - Two additionnal data disks were added - As I said, this is the only running VM stored as pre allocated. Regards, -- Nicolas Ecarnot
Le 26/03/2013 12:17, Nicolas Ecarnot a écrit :
Le 25/03/2013 12:10, Nicolas Ecarnot a écrit :
Le 24/03/2013 09:53, Dafna Ron a écrit :
is the vm preallocated or thin provision disk type?
This VM has 3 disks : - first disk to host the windows system : Thin provision - second disk to store some data : Preallocated - third disk to store some more data : Thin provision
I'm realizing that amongst the 15 VMs, only this one and another one that is stopped are using preallocated disks. I'm regularly migrating some VMs (and stopping and starting and playing with them) with no issue, and they all are using thin provisioned disks!
Could this be a common factor of the problem?
also, can you please attach engine, vdsm, libvirt and the vm's qemu logs?
Relevant logs :
############
Ok, I'm in the process of collecting the logs and posting them in a useable manner.
More to come.
Ok, once again, I ran a test and observed the relevant logs. I tried to isolate the time frames, but it may be long for vdsm.log
Here they are : * /var/log/libvirt/qemu/serv-chk-adm3.log http://pastebin.com/JVKMSmxD * /var/log/libvirtd.log http://pastebin.com/sWGDCqNh * /var/log/vdsm/vdsm.log (the BIG one) http://pastebin.com/bevTEhym
What I can add to help you help me, is that : - I saw that all my VM appear as tainted. I did not know what that meant (but RTFMed since), and this does not appear to disturb the other VMs - Many VMs including the problematic one have been imported from ovirt-v2v with now such issue. - This particular VM was also imported, but the starting point was a vmdk or ova single file. - Two additionnal data disks were added - As I said, this is the only running VM stored as pre allocated.
Regards,
One suggestion : I see no obvious errors in the log files. Could this paused state happen due to a VM's kernel panic? -- Nicolas Ecarnot
On 03/27/2013 10:38 AM, Nicolas Ecarnot wrote:
Le 26/03/2013 12:17, Nicolas Ecarnot a écrit :
Le 25/03/2013 12:10, Nicolas Ecarnot a écrit :
Le 24/03/2013 09:53, Dafna Ron a écrit :
is the vm preallocated or thin provision disk type?
This VM has 3 disks : - first disk to host the windows system : Thin provision - second disk to store some data : Preallocated - third disk to store some more data : Thin provision
I'm realizing that amongst the 15 VMs, only this one and another one that is stopped are using preallocated disks. I'm regularly migrating some VMs (and stopping and starting and playing with them) with no issue, and they all are using thin provisioned disks!
Could this be a common factor of the problem?
also, can you please attach engine, vdsm, libvirt and the vm's qemu logs?
Relevant logs :
############
Ok, I'm in the process of collecting the logs and posting them in a useable manner.
More to come.
Ok, once again, I ran a test and observed the relevant logs. I tried to isolate the time frames, but it may be long for vdsm.log
Here they are : * /var/log/libvirt/qemu/serv-chk-adm3.log http://pastebin.com/JVKMSmxD * /var/log/libvirtd.log http://pastebin.com/sWGDCqNh * /var/log/vdsm/vdsm.log (the BIG one) http://pastebin.com/bevTEhym
What I can add to help you help me, is that : - I saw that all my VM appear as tainted. I did not know what that meant (but RTFMed since), and this does not appear to disturb the other VMs - Many VMs including the problematic one have been imported from ovirt-v2v with now such issue. - This particular VM was also imported, but the starting point was a vmdk or ova single file. - Two additionnal data disks were added - As I said, this is the only running VM stored as pre allocated.
Regards,
One suggestion : I see no obvious errors in the log files. Could this paused state happen due to a VM's kernel panic?
is this still relevant?
Le 20/04/2013 22:55, Itamar Heim a écrit :
On 03/27/2013 10:38 AM, Nicolas Ecarnot wrote:
Le 26/03/2013 12:17, Nicolas Ecarnot a écrit :
Le 25/03/2013 12:10, Nicolas Ecarnot a écrit :
Le 24/03/2013 09:53, Dafna Ron a écrit :
is the vm preallocated or thin provision disk type?
This VM has 3 disks : - first disk to host the windows system : Thin provision - second disk to store some data : Preallocated - third disk to store some more data : Thin provision
I'm realizing that amongst the 15 VMs, only this one and another one that is stopped are using preallocated disks. I'm regularly migrating some VMs (and stopping and starting and playing with them) with no issue, and they all are using thin provisioned disks!
Could this be a common factor of the problem?
also, can you please attach engine, vdsm, libvirt and the vm's qemu logs?
Relevant logs :
############
Ok, I'm in the process of collecting the logs and posting them in a useable manner.
More to come.
Ok, once again, I ran a test and observed the relevant logs. I tried to isolate the time frames, but it may be long for vdsm.log
Here they are : * /var/log/libvirt/qemu/serv-chk-adm3.log http://pastebin.com/JVKMSmxD * /var/log/libvirtd.log http://pastebin.com/sWGDCqNh * /var/log/vdsm/vdsm.log (the BIG one) http://pastebin.com/bevTEhym
What I can add to help you help me, is that : - I saw that all my VM appear as tainted. I did not know what that meant (but RTFMed since), and this does not appear to disturb the other VMs - Many VMs including the problematic one have been imported from ovirt-v2v with now such issue. - This particular VM was also imported, but the starting point was a vmdk or ova single file. - Two additionnal data disks were added - As I said, this is the only running VM stored as pre allocated.
Regards,
One suggestion : I see no obvious errors in the log files. Could this paused state happen due to a VM's kernel panic?
is this still relevant?
It is! Further investigations from my colleague shown the following facts : - This VM has 3 disks. Only one of those disks is responsible for the problem - In this disk, my coworker has found only 3 files (database files) that he can do nothing with without leading to the freeze. - He tried to cat them into /dev/null, and this is leading to the freeze - He tried to copy them into another disk -> freeze! We see absolutely no evidence of a kernel panic. Rather, this seems to be related to a network bottleneck between the node and the iSCSI SAN, leading to oVirt unable to sustain a sufficent bandwidth and freezing the VM. Since then, we moved to another solution, but for the sake of opensource debugging, we did kept the faulty VM for your eyes only :) -- Nicolas Ecarnot
Nicolas Ecarnot <nicolas@...> writes:
Le 20/04/2013 22:55, Itamar Heim a écrit :
On 03/27/2013 10:38 AM, Nicolas Ecarnot wrote:
Le 26/03/2013 12:17, Nicolas Ecarnot a écrit :
Le 25/03/2013 12:10, Nicolas Ecarnot a écrit :
Le 24/03/2013 09:53, Dafna Ron a écrit :
is the vm preallocated or thin provision disk type?
This VM has 3 disks : - first disk to host the windows system : Thin provision - second disk to store some data : Preallocated - third disk to store some more data : Thin provision
I'm realizing that amongst the 15 VMs, only this one and another one that is stopped are using preallocated disks. I'm regularly migrating some VMs (and stopping and starting and
playing
with them) with no issue, and they all are using thin provisioned disks!
Could this be a common factor of the problem?
also, can you please attach engine, vdsm, libvirt and the vm's qemu logs?
Relevant logs :
############
Ok, I'm in the process of collecting the logs and posting them in a useable manner.
More to come.
Ok, once again, I ran a test and observed the relevant logs. I tried to isolate the time frames, but it may be long for vdsm.log
Here they are : * /var/log/libvirt/qemu/serv-chk-adm3.log http://pastebin.com/JVKMSmxD * /var/log/libvirtd.log http://pastebin.com/sWGDCqNh * /var/log/vdsm/vdsm.log (the BIG one) http://pastebin.com/bevTEhym
What I can add to help you help me, is that : - I saw that all my VM appear as tainted. I did not know what that meant (but RTFMed since), and this does not appear to disturb the other VMs - Many VMs including the problematic one have been imported from ovirt-v2v with now such issue. - This particular VM was also imported, but the starting point was a vmdk or ova single file. - Two additionnal data disks were added - As I said, this is the only running VM stored as pre allocated.
Regards,
One suggestion : I see no obvious errors in the log files. Could this paused state happen due to a VM's kernel panic?
is this still relevant?
It is! Further investigations from my colleague shown the following facts : - This VM has 3 disks. Only one of those disks is responsible for the problem - In this disk, my coworker has found only 3 files (database files) that he can do nothing with without leading to the freeze. - He tried to cat them into /dev/null, and this is leading to the freeze - He tried to copy them into another disk -> freeze!
We see absolutely no evidence of a kernel panic. Rather, this seems to be related to a network bottleneck between the node and the iSCSI SAN, leading to oVirt unable to sustain a sufficent bandwidth and freezing the VM.
Since then, we moved to another solution, but for the sake of opensource debugging, we did kept the faulty VM for your eyes only :)
Hi, anyone has answer? I got same issue. I create new vm and cannot start on ovirt node. The status wating for launch long time and vm cannot start. here is vdsm log: VM Channels Listener::DEBUG::2013-06-14 18:32:32,504::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:32,504::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/qemu/channels/4000570- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 32. VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::guestIF::95::vm.Vm:: (_connect) vmId=`187f61c9-d81f-491a-b5f0-4798ec6c8342`::Attempting connection to /var/lib/libvirt/qemu/channels/4000565- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 33. VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::guestIF::95::vm.Vm:: (_connect) vmId=`6c3074ae-c752-4622-94e7-a4ca09b252f7`::Attempting connection to /var/lib/libvirt/qemu/channels/4000563- 02.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 35. VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::guestIF::95::vm.Vm:: (_connect) vmId=`20f144cd-f027-4710-a433-dcdc62eec554`::Attempting connection to /var/lib/libvirt/qemu/channels/4000568- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/qemu/channels/4000570- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,507::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 32. VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::guestIF::95::vm.Vm:: (_connect) vmId=`187f61c9-d81f-491a-b5f0-4798ec6c8342`::Attempting connection to /var/lib/libvirt/qemu/channels/4000565- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 33. VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::guestIF::95::vm.Vm:: (_connect) vmId=`6c3074ae-c752-4622-94e7-a4ca09b252f7`::Attempting connection to /var/lib/libvirt/qemu/channels/4000563- 02.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 35. VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::guestIF::95::vm.Vm:: (_connect) vmId=`20f144cd-f027-4710-a433-dcdc62eec554`::Attempting connection to /var/lib/libvirt/qemu/channels/4000568- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/q
Hi I add a similar issue on fedora18, I fixed it : yum update --enablerepo=updates-testing systemd-197-1.fc18.2 Do know if it can help. Kevin 2013/6/14 Cuongds <cuongds.hut@gmail.com>
Nicolas Ecarnot <nicolas@...> writes:
Le 20/04/2013 22:55, Itamar Heim a écrit :
On 03/27/2013 10:38 AM, Nicolas Ecarnot wrote:
Le 26/03/2013 12:17, Nicolas Ecarnot a écrit :
Le 25/03/2013 12:10, Nicolas Ecarnot a écrit :
Le 24/03/2013 09:53, Dafna Ron a écrit : > is the vm preallocated or thin provision disk type?
This VM has 3 disks : - first disk to host the windows system : Thin provision - second disk to store some data : Preallocated - third disk to store some more data : Thin provision
I'm realizing that amongst the 15 VMs, only this one and another one that is stopped are using preallocated disks. I'm regularly migrating some VMs (and stopping and starting and
playing
with them) with no issue, and they all are using thin provisioned disks!
Could this be a common factor of the problem?
> > also, can you please attach engine, vdsm, libvirt and the vm's qemu > logs?
Relevant logs :
############
Ok, I'm in the process of collecting the logs and posting them in a useable manner.
More to come.
Ok, once again, I ran a test and observed the relevant logs. I tried to isolate the time frames, but it may be long for vdsm.log
Here they are : * /var/log/libvirt/qemu/serv-chk-adm3.log http://pastebin.com/JVKMSmxD * /var/log/libvirtd.log http://pastebin.com/sWGDCqNh * /var/log/vdsm/vdsm.log (the BIG one) http://pastebin.com/bevTEhym
What I can add to help you help me, is that : - I saw that all my VM appear as tainted. I did not know what that meant (but RTFMed since), and this does not appear to disturb the other VMs - Many VMs including the problematic one have been imported from ovirt-v2v with now such issue. - This particular VM was also imported, but the starting point was a vmdk or ova single file. - Two additionnal data disks were added - As I said, this is the only running VM stored as pre allocated.
Regards,
One suggestion : I see no obvious errors in the log files. Could this paused state happen due to a VM's kernel panic?
is this still relevant?
It is! Further investigations from my colleague shown the following facts : - This VM has 3 disks. Only one of those disks is responsible for the problem - In this disk, my coworker has found only 3 files (database files) that he can do nothing with without leading to the freeze. - He tried to cat them into /dev/null, and this is leading to the freeze - He tried to copy them into another disk -> freeze!
We see absolutely no evidence of a kernel panic. Rather, this seems to be related to a network bottleneck between the node and the iSCSI SAN, leading to oVirt unable to sustain a sufficent bandwidth and freezing the VM.
Since then, we moved to another solution, but for the sake of opensource debugging, we did kept the faulty VM for your eyes only :)
Hi, anyone has answer? I got same issue. I create new vm and cannot start on ovirt node. The status wating for launch long time and vm cannot start. here is vdsm log: VM Channels Listener::DEBUG::2013-06-14 18:32:32,504::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:32,504::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/qemu/channels/4000570- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 32. VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::guestIF::95::vm.Vm:: (_connect) vmId=`187f61c9-d81f-491a-b5f0-4798ec6c8342`::Attempting connection to /var/lib/libvirt/qemu/channels/4000565- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 33. VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::guestIF::95::vm.Vm:: (_connect) vmId=`6c3074ae-c752-4622-94e7-a4ca09b252f7`::Attempting connection to /var/lib/libvirt/qemu/channels/4000563- 02.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 35. VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::guestIF::95::vm.Vm:: (_connect) vmId=`20f144cd-f027-4710-a433-dcdc62eec554`::Attempting connection to /var/lib/libvirt/qemu/channels/4000568- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/qemu/channels/4000570- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,507::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 32. VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::guestIF::95::vm.Vm:: (_connect) vmId=`187f61c9-d81f-491a-b5f0-4798ec6c8342`::Attempting connection to /var/lib/libvirt/qemu/channels/4000565- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 33. VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::guestIF::95::vm.Vm:: (_connect) vmId=`6c3074ae-c752-4622-94e7-a4ca09b252f7`::Attempting connection to /var/lib/libvirt/qemu/channels/4000563- 02.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 35. VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::guestIF::95::vm.Vm:: (_connect) vmId=`20f144cd-f027-4710-a433-dcdc62eec554`::Attempting connection to /var/lib/libvirt/qemu/channels/4000568- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/q
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Kevin Mazière Responsable Infrastructure Alter Way – Hosting 1 rue Royal - 227 Bureaux de la Colline 92213 Saint-Cloud Cedex Tél : +33 (0)1 41 16 38 41 Mob : +33 (0)7 62 55 57 05 http://www.alterway.fr
On 06/14/2013 06:55 PM, Kevin Maziere Aubry wrote:
Hi
I add a similar issue on fedora18, I fixed it : yum update --enablerepo=updates-testing systemd-197-1.fc18.2 Do know if it can help.
while requires all hosts in the cluster to be the same, please check if reproduces with an updated .el6 host (RHEL/CentOS)?
Kevin
2013/6/14 Cuongds <cuongds.hut@gmail.com <mailto:cuongds.hut@gmail.com>>
Nicolas Ecarnot <nicolas@...> writes:
> > Le 20/04/2013 22:55, Itamar Heim a écrit : > > On 03/27/2013 10:38 AM, Nicolas Ecarnot wrote: > >> Le 26/03/2013 12:17, Nicolas Ecarnot a écrit : > >>> Le 25/03/2013 12:10, Nicolas Ecarnot a écrit : > >>>> Le 24/03/2013 09:53, Dafna Ron a écrit : > >>>>> is the vm preallocated or thin provision disk type? > >>>> > >>>> This VM has 3 disks : > >>>> - first disk to host the windows system : Thin provision > >>>> - second disk to store some data : Preallocated > >>>> - third disk to store some more data : Thin provision > >>>> > >>>> I'm realizing that amongst the 15 VMs, only this one and another one > >>>> that is stopped are using preallocated disks. > >>>> I'm regularly migrating some VMs (and stopping and starting and playing > >>>> with them) with no issue, and they all are using thin provisioned > >>>> disks! > >>>> > >>>> Could this be a common factor of the problem? > >>>> > >>>>> > >>>>> also, can you please attach engine, vdsm, libvirt and the vm's qemu > >>>>> logs? > >>>> > >>>> Relevant logs : > >>>> > >>>> ############ > >>>> > >>>> Ok, I'm in the process of collecting the logs and posting them in a > >>>> useable manner. > >>>> > >>>> More to come. > >>> > >>> Ok, once again, I ran a test and observed the relevant logs. > >>> I tried to isolate the time frames, but it may be long for vdsm.log > >>> > >>> Here they are : > >>> * /var/log/libvirt/qemu/serv-chk-adm3.log > >>> http://pastebin.com/JVKMSmxD > >>> * /var/log/libvirtd.log > >>> http://pastebin.com/sWGDCqNh > >>> * /var/log/vdsm/vdsm.log (the BIG one) > >>> http://pastebin.com/bevTEhym > >>> > >>> What I can add to help you help me, is that : > >>> - I saw that all my VM appear as tainted. I did not know what that meant > >>> (but RTFMed since), and this does not appear to disturb the other VMs > >>> - Many VMs including the problematic one have been imported from > >>> ovirt-v2v with now such issue. > >>> - This particular VM was also imported, but the starting point was a > >>> vmdk or ova single file. > >>> - Two additionnal data disks were added > >>> - As I said, this is the only running VM stored as pre allocated. > >>> > >>> Regards, > >>> > >> > >> One suggestion : I see no obvious errors in the log files. Could this > >> paused state happen due to a VM's kernel panic? > >> > > > > is this still relevant? > > It is! > Further investigations from my colleague shown the following facts : > - This VM has 3 disks. Only one of those disks is responsible for the > problem > - In this disk, my coworker has found only 3 files (database files) that > he can do nothing with without leading to the freeze. > - He tried to cat them into /dev/null, and this is leading to the freeze > - He tried to copy them into another disk -> freeze! > > We see absolutely no evidence of a kernel panic. > Rather, this seems to be related to a network bottleneck between the > node and the iSCSI SAN, leading to oVirt unable to sustain a sufficent > bandwidth and freezing the VM. > > Since then, we moved to another solution, but for the sake of opensource > debugging, we did kept the faulty VM for your eyes only :) >
Hi, anyone has answer? I got same issue. I create new vm and cannot start on ovirt node. The status wating for launch long time and vm cannot start. here is vdsm log: VM Channels Listener::DEBUG::2013-06-14 18:32:32,504::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:32,504::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/qemu/channels/4000570- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 32. VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::guestIF::95::vm.Vm:: (_connect) vmId=`187f61c9-d81f-491a-b5f0-4798ec6c8342`::Attempting connection to /var/lib/libvirt/qemu/channels/4000565- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 33. VM Channels Listener::DEBUG::2013-06-14 18:32:33,505::guestIF::95::vm.Vm:: (_connect) vmId=`6c3074ae-c752-4622-94e7-a4ca09b252f7`::Attempting connection to /var/lib/libvirt/qemu/channels/4000563- 02.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 35. VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::guestIF::95::vm.Vm:: (_connect) vmId=`20f144cd-f027-4710-a433-dcdc62eec554`::Attempting connection to /var/lib/libvirt/qemu/channels/4000568- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:33,506::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/qemu/channels/4000570- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,507::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 32. VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::guestIF::95::vm.Vm:: (_connect) vmId=`187f61c9-d81f-491a-b5f0-4798ec6c8342`::Attempting connection to /var/lib/libvirt/qemu/channels/4000565- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 33. VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::guestIF::95::vm.Vm:: (_connect) vmId=`6c3074ae-c752-4622-94e7-a4ca09b252f7`::Attempting connection to /var/lib/libvirt/qemu/channels/4000563- 02.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,508::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 35. VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::guestIF::95::vm.Vm:: (_connect) vmId=`20f144cd-f027-4710-a433-dcdc62eec554`::Attempting connection to /var/lib/libvirt/qemu/channels/4000568- 01.com.redhat.rhevm.vdsm VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::vmChannels::104::vds:: (_handle_unconnected) Trying to connect fileno 37. VM Channels Listener::DEBUG::2013-06-14 18:32:34,509::guestIF::95::vm.Vm:: (_connect) vmId=`ffd60b1c-9a3c-4853-88aa-7973f9756c96`::Attempting connection to /var/lib/libvirt/q
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--
Kevin Mazière Responsable Infrastructure Alter Way – Hosting 1 rue Royal - 227 Bureaux de la Colline 92213 Saint-Cloud Cedex Tél : +33 (0)1 41 16 38 41 Mob : +33 (0)7 62 55 57 05 http://www.alterway.fr <http://www.alterway.fr/>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (6)
-
Cuongds -
Dafna Ron -
Itamar Heim -
Jaco -
Kevin Maziere Aubry -
Nicolas Ecarnot