[Users] Gluster VM stuck in "waiting for launch" state

Wed Oct 30 19:41:43 UTC 2013

Il 30/10/2013 18:04, Dan Kenigsberg ha scritto:
> On Wed, Oct 30, 2013 at 02:40:02PM +0100, Alessandro Bianchi wrote:
>>     Il 30/10/2013 13:58, Dan Kenigsberg ha scritto:
>>
>>   On Wed, Oct 30, 2013 at 11:34:21AM +0100, Alessandro Bianchi wrote:
>>
>>      Hi everyone
>>
>>      I've set up a gluster storage with two replicated bricks
>>
>>      DC is up and I created a VM to test gluster storage
>>
>>      If I start the VM WITHOUT any disk attached (only one virtual DVD) it
>>      starts fine.
>>
>>      If I attach a gluster domain disk thin provisioning 30 Gb the Vm stucks in
>>      "waiting for launch" state
>>
>>      I see no special activity on the gluster servers (they serve several other
>>      shares with no troubles at all and even the ISO domain  is a NFS on
>>      locally mounted gluster and works fine)
>>
>>      I've double checked all the pre requisites and they look fine (F 19 -
>>      gluster setup insecure  in both glusterd.vol and volume options -
>>      uid/gid/insecure )
>>
>>      Am I doing something wrong?
>>
>>      I'm even unable to stop the VM from the engine GUI
>>
>>      Any advise?
>>
>>   Which version of ovirt are you using? Hopefully ovirt-3.3.0.1.
>>   For how long is the VM stuck in its "wait for launch" state?
>>   What does `virsh -r list` has to say while startup stalls?
>>   Would you provide more content of your vdsm.log and possibly
>>   libvirtd.log so we can understand what blocks the VM start-up? Please
>>   use attachement of pastebin, as your mail agents wreaks havoc to the log
>>   lines.
>>
>>
>>     Thank you for your answer.
>>
>>     Here are the "facts"
>>
>>     In the GUI I see
>>
>>     "waiting for launch 3 h"
>>
>>      virsh -r list
>>      Id    Nome                           Stato
>>     ----------------------------------------------------
>>      3     CentOS_30                      terminato
>>
>>     vdsClient -s 0 list table
>>     200dfb05-461e-49d9-95a2-c0a7c7ced669      0  CentOS_30
>>     WaitForLaunch
>>
>>     Packages:
>>
>>     ovirt-engine-userportal-3.3.0.1-1.fc19.noarch
>>     ovirt-log-collector-3.3.1-1.fc19.noarch
>>     ovirt-engine-restapi-3.3.0.1-1.fc19.noarch
>>     ovirt-engine-setup-3.3.0.1-1.fc19.noarch
>>     ovirt-engine-backend-3.3.0.1-1.fc19.noarch
>>     ovirt-host-deploy-java-1.1.1-1.fc19.noarch
>>     ovirt-release-fedora-8-1.noarch
>>     ovirt-engine-setup-plugin-allinone-3.3.0.1-1.fc19.noarch
>>     ovirt-engine-webadmin-portal-3.3.0.1-1.fc19.noarch
>>     ovirt-engine-sdk-python-3.3.0.7-1.fc19.noarch
>>     ovirt-iso-uploader-3.3.1-1.fc19.noarch
>>     ovirt-engine-websocket-proxy-3.3.0.1-1.fc19.noarch
>>     ovirt-engine-dbscripts-3.3.0.1-1.fc19.noarch
>>     ovirt-host-deploy-offline-1.1.1-1.fc19.noarch
>>     ovirt-engine-cli-3.3.0.5-1.fc19.noarch
>>     ovirt-engine-tools-3.3.0.1-1.fc19.noarch
>>     ovirt-engine-lib-3.3.0.1-1.fc19.noarch
>>     ovirt-image-uploader-3.3.1-1.fc19.noarch
>>     ovirt-engine-3.3.0.1-1.fc19.noarch
>>     ovirt-host-deploy-1.1.1-1.fc19.noarch
>>
>>     I attach the full vdsm log
>>
>>     Look around 30-10 10:30 to see all what happens
>>
>>     Despite the "terminated" label in output from virsh I still see the VM
>>     "waiting for launch" in the GUI, so I suspect the answer to "how long" may
>>     be "forever"
>>
>>     Since this is a test VM I can do whatever test you may need to track the
>>     problem included destroy and rebuild
>>
>>     It would be great to have gluster support stable in ovirt!
>>
>>     Thank you for your efforts
> The log has an ominous failed attempt to start the VM, followed by an
> immediate vdsm crash. Is it reproducible?
>
> We have plenty of issues lurking here:
> 1. Why has libvirt failed to create the VM? For this, please find clues
>     in the complete non-line-broken CentOS_30.log and libvirtd.log.
attached to this messages
>
> 2. Why was vdsm killed? Does /var/log/message has a clue from systemd?
result of cat /var/log/messages | grep vdsm attached
>
> 3. We may have a nasty race: if Vdsm crashes just before it has
>     registered that the VM is down.
>
> 4. We used to force Vdsm to run with LC_ALL=C. It seems that the grand
>     service rewrite by Zhou (http://gerrit.ovirt.org/15578) has changed
>     that. This may have adverse effects, since AFAIR we sometimes parse
>     application output, and assume that it's in C. Having a non-English
>     log file is problematic on it's own for support personal, used to
>     grep for keywords. ybronhei, was it intensional? Can it be reverted
>     or at least scrutinized?
currentely it still says "waiting for launch 9h"

I don't abort it so if you need any other info I can have them

Thgank you
>
> Thread-77::ERROR::2013-10-30 08:51:13,147::vm::2062::vm.Vm::(_startUnderlyingVm) vmId=`73e6615b-78c3-42e5-803a-3fc20d64ca32`::The vm start process failed
> Traceback (most recent call last):
>    File "/usr/share/vdsm/vm.py", line 2022, in _startUnderlyingVm
>      self._run()
>    File "/usr/share/vdsm/vm.py", line 2906, in _run
>      self._connection.createXML(domxml, flags),
>    File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
>      ret = f(*args, **kwargs)
>    File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2805, in createXML
>      if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
> libvirtError: Unable to read from monitor: Connessione interrotta dal corrispondente
> Thread-77::DEBUG::2013-10-30 08:51:13,151::vm::2448::vm.Vm::(setDownStatus) vmId=`73e6615b-78c3-42e5-803a-3fc20d64ca32`::Changed state to Down: Unable to read from monitor: Connessione interrotta dal corrispondente
> MainThread::DEBUG::2013-10-30 08:51:13,153::vdsm::45::vds::(sigtermHandler) Received signal 15
> MainThread::DEBUG::2013-10-30 08:51:13,153::clientIF::210::vds::(prepareForShutdown) cannot run prepareForShutdown twice
> MainThread::DEBUG::2013-10-30 08:51:15,633::vdsm::45::vds::(sigtermHandler) Received signal 15
> MainThread::DEBUG::2013-10-30 08:51:15,633::clientIF::210::vds::(prepareForShutdown) cannot run prepareForShutdown twice
> MainThread::INFO::2013-10-30 08:51:15,700::vdsm::101::vds::(run) (PID: 7726) I am the actual vdsm 4.12.1-4.fc19 hypervisor.skynet.it (3.11.1-200.fc19.x86_64)
>

-- 

SkyNet SRL

Via Maggiate 67/a - 28021 Borgomanero (NO) - tel. +39 0322-836487/834765 
- fax +39 0322-836608

http://www.skynet.it <http://www.skynet.it/>

Autorizzazione Ministeriale n.197

Le informazioni contenute in questo messaggio sono riservate e 
confidenziali ed è vietata la diffusione in qualunque modo eseguita.
Qualora Lei non fosse la persona a cui il presente messaggio è 
destinato, La invitiamo ad eliminarlo ed a distruggerlo non 
divulgandolo, dandocene gentilmente comunicazione.
Per qualsiasi informazione si prega di contattare info at skynet.it (e-mail 
dell'azienda). Rif. D.L. 196/2003

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20131030/ddaa58ff/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: CentOS_30.log.tar.xz
Type: application/x-xz
Size: 2640 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20131030/ddaa58ff/attachment-0003.xz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libvirtd.log.tar.xz
Type: application/x-xz
Size: 335476 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20131030/ddaa58ff/attachment-0004.xz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: messages.tar.xz
Type: application/x-xz
Size: 24004 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20131030/ddaa58ff/attachment-0005.xz>