<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body style="font-family: sans-serif; font-size: 16px;"
bgcolor="#FFFFFF" text="#000000">
<div id="QCMcontainer"
style="font-family:sans-serif;font-size:16px;"><br>
<br>
<span style="color:#000000; font-family:sans-serif;
font-size:12px; font-weight:normal" class="headerSpan">
<div class="moz-cite-prefix">Il 31/10/2013 07:21, Sahina Bose ha
scritto:<br>
</div>
</span>
<blockquote style="border-left: 2px solid rgb(0, 153, 0) !
important; border-right: 2px solid rgb(0, 153, 0) ! important;
padding: 0px 15px; margin: 8px 2px; font-size: medium;"
cite="mid:5271F6EB.8090806@redhat.com" type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->
<br>
On 10/31/2013 09:22 AM, Zhou Zheng Sheng wrote:
<br>
<blockquote style="border-left: 2px solid #009900 !important;
border-right: 2px solid #009900 !important; padding: 0px 15px
0px 15px; margin: 8px 2px;" type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->
<br>
on 2013/10/31 06:24, Dan Kenigsberg wrote:
<br>
<blockquote style="border-left: 2px solid #009900 !important;
border-right: 2px solid #009900 !important; padding: 0px
15px 0px 15px; margin: 8px 2px;" type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->On
Wed, Oct 30, 2013 at 08:41:43PM +0100, Alessandro Bianchi
wrote:
<br>
<blockquote style="border-left: 2px solid #009900
!important; border-right: 2px solid #009900 !important;
padding: 0px 15px 0px 15px; margin: 8px 2px;" type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->Il
30/10/2013 18:04, Dan Kenigsberg ha scritto:
<br>
<blockquote style="border-left: 2px solid #009900
!important; border-right: 2px solid #009900 !important;
padding: 0px 15px 0px 15px; margin: 8px 2px;"
type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->On
Wed, Oct 30, 2013 at 02:40:02PM +0100, Alessandro
Bianchi wrote:
<br>
<blockquote style="border-left: 2px solid #009900
!important; border-right: 2px solid #009900
!important; padding: 0px 15px 0px 15px; margin: 8px
2px;" type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->
Il 30/10/2013 13:58, Dan Kenigsberg ha scritto:
<br>
<br>
On Wed, Oct 30, 2013 at 11:34:21AM +0100, Alessandro
Bianchi wrote:
<br>
<br>
Hi everyone
<br>
<br>
I've set up a gluster storage with two replicated
bricks
<br>
<br>
DC is up and I created a VM to test gluster
storage
<br>
<br>
If I start the VM WITHOUT any disk attached (only
one virtual DVD) it
<br>
starts fine.
<br>
<br>
If I attach a gluster domain disk thin
provisioning 30 Gb the Vm stucks in
<br>
"waiting for launch" state
<br>
<br>
I see no special activity on the gluster servers
(they serve several other
<br>
shares with no troubles at all and even the ISO
domain is a NFS on
<br>
locally mounted gluster and works fine)
<br>
<br>
I've double checked all the pre requisites and
they look fine (F 19 -
<br>
gluster setup insecure in both glusterd.vol and
volume options -
<br>
uid/gid/insecure )
<br>
<br>
Am I doing something wrong?
<br>
<br>
I'm even unable to stop the VM from the engine
GUI
<br>
<br>
Any advise?
<br>
<br>
Which version of ovirt are you using? Hopefully
ovirt-3.3.0.1.
<br>
For how long is the VM stuck in its "wait for
launch" state?
<br>
What does `virsh -r list` has to say while startup
stalls?
<br>
Would you provide more content of your vdsm.log and
possibly
<br>
libvirtd.log so we can understand what blocks the VM
start-up? Please
<br>
use attachement of pastebin, as your mail agents
wreaks havoc to the log
<br>
lines.
<br>
<br>
<br>
Thank you for your answer.
<br>
<br>
Here are the "facts"
<br>
<br>
In the GUI I see
<br>
<br>
"waiting for launch 3 h"
<br>
<br>
virsh -r list
<br>
Id Nome Stato
<br>
----------------------------------------------------
<br>
3 CentOS_30 terminato
<br>
<br>
vdsClient -s 0 list table
<br>
200dfb05-461e-49d9-95a2-c0a7c7ced669 0
CentOS_30
<br>
WaitForLaunch
<br>
<br>
Packages:
<br>
<br>
ovirt-engine-userportal-3.3.0.1-1.fc19.noarch
<br>
ovirt-log-collector-3.3.1-1.fc19.noarch
<br>
ovirt-engine-restapi-3.3.0.1-1.fc19.noarch
<br>
ovirt-engine-setup-3.3.0.1-1.fc19.noarch
<br>
ovirt-engine-backend-3.3.0.1-1.fc19.noarch
<br>
ovirt-host-deploy-java-1.1.1-1.fc19.noarch
<br>
ovirt-release-fedora-8-1.noarch
<br>
ovirt-engine-setup-plugin-allinone-3.3.0.1-1.fc19.noarch
<br>
ovirt-engine-webadmin-portal-3.3.0.1-1.fc19.noarch
<br>
ovirt-engine-sdk-python-3.3.0.7-1.fc19.noarch
<br>
ovirt-iso-uploader-3.3.1-1.fc19.noarch
<br>
ovirt-engine-websocket-proxy-3.3.0.1-1.fc19.noarch
<br>
ovirt-engine-dbscripts-3.3.0.1-1.fc19.noarch
<br>
ovirt-host-deploy-offline-1.1.1-1.fc19.noarch
<br>
ovirt-engine-cli-3.3.0.5-1.fc19.noarch
<br>
ovirt-engine-tools-3.3.0.1-1.fc19.noarch
<br>
ovirt-engine-lib-3.3.0.1-1.fc19.noarch
<br>
ovirt-image-uploader-3.3.1-1.fc19.noarch
<br>
ovirt-engine-3.3.0.1-1.fc19.noarch
<br>
ovirt-host-deploy-1.1.1-1.fc19.noarch
<br>
<br>
I attach the full vdsm log
<br>
<br>
Look around 30-10 10:30 to see all what happens
<br>
<br>
Despite the "terminated" label in output from
virsh I still see the VM
<br>
"waiting for launch" in the GUI, so I suspect the
answer to "how long" may
<br>
be "forever"
<br>
<br>
Since this is a test VM I can do whatever test you
may need to track the
<br>
problem included destroy and rebuild
<br>
<br>
It would be great to have gluster support stable
in ovirt!
<br>
<br>
Thank you for your efforts
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
The log has an ominous failed attempt to start the VM,
followed by an
<br>
immediate vdsm crash. Is it reproducible?
<br>
<br>
We have plenty of issues lurking here:
<br>
1. Why has libvirt failed to create the VM? For this,
please find clues
<br>
in the complete non-line-broken CentOS_30.log and
libvirtd.log.
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
attached to this messages
<br>
<blockquote style="border-left: 2px solid #009900
!important; border-right: 2px solid #009900 !important;
padding: 0px 15px 0px 15px; margin: 8px 2px;"
type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->2.
Why was vdsm killed? Does /var/log/message has a clue
from systemd?
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
result of cat /var/log/messages | grep vdsm attached
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
I do not see an explicit attempt to take vdsmd down. Do you
see any other
<br>
incriminating message correlated with
<br>
<br>
Oct 30 08:51:15 hypervisor respawn: slave
'/usr/share/vdsm/vdsm' died, respawning slave
<br>
<br>
<br>
<blockquote style="border-left: 2px solid #009900
!important; border-right: 2px solid #009900 !important;
padding: 0px 15px 0px 15px; margin: 8px 2px;" type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->
<blockquote style="border-left: 2px solid #009900
!important; border-right: 2px solid #009900 !important;
padding: 0px 15px 0px 15px; margin: 8px 2px;"
type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->3.
We may have a nasty race: if Vdsm crashes just before it
has
<br>
registered that the VM is down.
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
<!--[if !IE]></DIV><![endif]--></blockquote>
Actually, this is not the issue: vdsm tries (and fails, due
to qemu/libvirt
<br>
bug) to destroy the VM.
<br>
<br>
<blockquote style="border-left: 2px solid #009900
!important; border-right: 2px solid #009900 !important;
padding: 0px 15px 0px 15px; margin: 8px 2px;" type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->
<blockquote style="border-left: 2px solid #009900
!important; border-right: 2px solid #009900 !important;
padding: 0px 15px 0px 15px; margin: 8px 2px;"
type="cite"><!--[if !IE]><DIV style="border-left: 2px solid #009900; border-right: 2px solid #009900; padding: 0px 15px; margin: 2px 0px;"><![endif]-->4.
We used to force Vdsm to run with LC_ALL=C. It seems
that the grand
<br>
service rewrite by Zhou
(<a class="moz-txt-link-freetext" href="http://gerrit.ovirt.org/15578">http://gerrit.ovirt.org/15578</a>) has changed
<br>
that. This may have adverse effects, since AFAIR we
sometimes parse
<br>
application output, and assume that it's in C.
Having a non-English
<br>
log file is problematic on it's own for support
personal, used to
<br>
grep for keywords. ybronhei, was it intensional? Can
it be reverted
<br>
or at least scrutinized?
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
currentely it still says "waiting for launch 9h"
<br>
<br>
I don't abort it so if you need any other info I can have
them
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
libvirtd fails to connect to qemu's monitor. This smells
like a qemu bug that
<br>
is beyond my over-the-mailing-list debugging abilities :-(
<br>
You may want to strace or gdb the running qemu process, or
to try to re-attach
<br>
to it by restarting libvirtd.
<br>
<br>
2013-10-30 07:51:10.045+0000: 8304: debug :
qemuProcessStart:3804 : Waiting for monitor to show up
<br>
2013-10-30 07:51:10.045+0000: 8304: debug :
qemuProcessWaitForMonitor:1707 : Connect monitor to
0x7fc1640eab80 'CentOS_30'
<br>
2013-10-30 07:51:10.246+0000: 8304: debug :
qemuMonitorOpenInternal:751 : QEMU_MONITOR_NEW:
mon=0x7fc1640ef6b0 refs=2 fd=27
<br>
2013-10-30 07:51:10.246+0000: 8304: debug :
qemuMonitorSetCapabilities:1145 : mon=0x7fc1640ef6b0
<br>
2013-10-30 07:51:10.246+0000: 8304: debug :
qemuMonitorSend:887 : QEMU_MONITOR_SEND_MSG:
mon=0x7fc1640ef6b0
msg={"execute":"qmp_capabilities","id":"libvirt-1"}
<br>
<br>
fd=-1
<br>
2013-10-30 07:51:13.097+0000: 8296: error :
qemuMonitorIORead:505 : Unable to read from monitor:
Connessione interrotta dal corrispondente
<br>
2013-10-30 07:51:13.097+0000: 8296: debug :
qemuMonitorIO:638 : Error on monitor Unable to read from
monitor: Connessione interrotta dal corrispondente
<br>
2013-10-30 07:51:13.097+0000: 8296: debug :
qemuMonitorIO:672 : Triggering error callback
<br>
2013-10-30 07:51:13.097+0000: 8296: debug :
qemuProcessHandleMonitorError:351 : Received error on
0x7fc1640eab80 'CentOS_30'
<br>
2013-10-30 07:51:13.097+0000: 8304: debug :
qemuMonitorSend:899 : Send command resulted in error Unable
to read from monitor: Connessione interrotta dal
corrispondente
<br>
2013-10-30 07:51:13.097+0000: 8304: debug :
qemuProcessStop:3992 : Shutting down VM 'CentOS_30' pid=7655
flags=0
<br>
2013-10-30 07:51:13.097+0000: 8296: debug :
qemuMonitorIO:638 : Error on monitor Unable to read from
monitor: Connessione interrotta dal corrispondente
<br>
2013-10-30 07:51:13.097+0000: 8296: debug :
qemuMonitorIO:661 : Triggering EOF callback
<br>
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
I noticed in the previous attached CentOS_30.log, qemu said
"Gluster
<br>
connection failed for server=172.16.0.100" for each of the
first few
<br>
attempts to start qemu. That's why libvirt said it could not
connect to
<br>
qemu monitor, the real reason was that qemu process exited
after it
<br>
checked the gluster storage. libvirt error message was not
very friendly
<br>
in this case.
<br>
<br>
Then I see on "2013-10-30 08:28:51.047+0000" in CentOS_30.log,
qemu
<br>
started successfully without reporting gluster error, this was
fine. In
<br>
libvirtd.log on the corresponding time spot, it libvirt
connected to the
<br>
monitor and reported it was about to issue a handshake.
<br>
<br>
Investigate the CentOS_30.log further, I find the attempts to
start qemu
<br>
after "2013-10-30 08:28:51" were failed with the same error
"Gluster
<br>
connection failed for server=172.16.0.100". Maybe there is a
problem in
<br>
the gluster server or firewall? I'm not sure.
<br>
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
<br>
<a class="moz-txt-link-freetext" href="http://www.ovirt.org/Features/GlusterFS_Storage_Domain#Important_Pre-requisites">http://www.ovirt.org/Features/GlusterFS_Storage_Domain#Important_Pre-requisites</a>
- I assume these have been followed w.r.t setting permissions.
<br>
<br>
<br>
<br>
<!--[if !IE]></DIV><![endif]--></blockquote>
<br>
Yes have been followed for both glusterd,vol AND volume<br>
<div class="moz-signature">-- <br>
<meta http-equiv="CONTENT-TYPE" content="text/html;
charset=UTF-8">
<title></title>
<meta name="generator" content="Bluefish 2.0.3">
<meta name="author" content="Alessandro Bianchi">
<meta name="CREATED" content="20100306;9474300">
<meta name="CHANGEDBY" content="Alessandro ">
<meta name="CHANGED" content="20100306;10212100">
<style type="text/css">
<!--
P { font-family: "Arial", "Helvetica", sans-serif; font-size: 10pt }
P.nome { color: #ff8000; font-family: "Arial", "Helvetica", sans-serif; font-size: 12pt; font-weight: bold; text-align: center }
P.indirizzo { color: #0084d1; font-family: "Arial", "Helvetica", sans-serif; font-size: 10pt; font-weight: bold; line-height: 0.48cm; text-align: center }
P.info { color: #b3b3b3; font-family: "Arial", "Helvetica", sans-serif; font-size: 9pt }
A:link { color: #005dff; text-decoration: none }
A:visited { color: #005dff; text-decoration: none }
-->
</style>
<p class="nome">SkyNet SRL</p>
<p class="indirizzo">Via Maggiate 67/a - 28021 Borgomanero (NO)
- tel.
+39 0322-836487/834765 - fax +39 0322-836608</p>
<p align="CENTER"><a href="http://www.skynet.it/">http://www.skynet.it</a></p>
<p class="indirizzo">Autorizzazione Ministeriale n.197</p>
<p class="info">Le informazioni contenute in questo messaggio
sono
riservate e confidenziali ed è vietata la diffusione in
qualunque
modo eseguita.<br>
Qualora Lei non fosse la persona a cui il presente
messaggio è destinato, La invitiamo ad eliminarlo ed a
distruggerlo
non divulgandolo, dandocene gentilmente comunicazione. <br>
Per
qualsiasi informazione si prega di contattare <a class="moz-txt-link-abbreviated" href="mailto:info@skynet.it">info@skynet.it</a>
(e-mail dell'azienda). Rif. D.L. 196/2003</p>
</div>
</div>
</body>
</html>