9:31 a.m.
This is a multi-part message in MIME format.
--------------FB3CCABEFD28F954C51E2EC1
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Hi,
anything in the journal about libvirt? (journalctl -u libvirtd)
could you share a bigger chunk of the vdsm log, demonstrating the failed
VM start?
Bests,
On 10/30/2017 03:28 PM, Hristo Pavlov wrote:
I tried it already, doesn't start on any of the nodes.
In a log /var/log/libvirt/qemu/HostedEngine.log at allthree nodes has
nothing to do with starting, as if it had not started.
Понедельник, 30 октября 2017, 16:11 +02:00 от Nathanaël Blanchet
<blanchet(a)abes.fr>:
Hi,
It happened to me too, after a live migration of it, I shut down
the hosted engine on the targeted host, and I couldn't restart it
anymore on this specific host. But I was able to start it on the
initial one, where I initially where I deployed the HE. It was
like the lease and the libvirt host definition staid on the first
host after migration.
So try you may try hosted-engine --vm-start on one of other hosts...
Le 30/10/2017 à 12:28, Hristo Pavlov a écrit :
> Hi All,
>
> Our oVirt cluster is with 3 nodes with shared fibre channel
> storage, the engine virtual machine is self hosted.
>
> Hypervisors OS: CentOS Linux release 7.3 / x86_64, oVirt version
> is 4.1.2.2. The environment has been working for about a year
> without any problems.
>
> Aftershutdown of the hosted engine virtual machine, it doesn't
> start.
>
> Тhese commands that were executed:
>
> hosted-engine --set-maintenance --mode=global
> hosted-engine --vm-shutdown
>
> after the status of engine vm was down, we executed start.
>
> [root@alpha] hosted-engine --vm-start
> VM exists and is down, destroying it
> Exception in thread Client localhost:54321 (most likely raised
> during interpreter shutdown):
>
> we noticied that at vdsm.log
>
> 017-10-30 13:11:04,863+0200 INFO (jsonrpc/1)
> [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats succeeded
> in 0.26 seconds (__init__:533)
> 2017-10-30 13:11:05,802+0200 INFO (jsonrpc/6)
> [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in
> 0.01 seconds (__init__:533)
> 2017-10-30 13:11:05,825+0200 WARN (jsonrpc/2) [virt.vm]
> (vmId='da98112d-b9fb-4098-93fa-1f1374b41e46') Failed to get
> metadata, domain not connected. (vm:2765)
> 2017-10-30 13:11:05,825+0200 ERROR (jsonrpc/2)
> [jsonrpc.JsonRpcServer] Internal server error (__init__:570)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
> line 565, in _handle_request
> res = method(**params)
> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py",
> line 202, in _dynamicMethod
> result = fn(*methodArgs)
> File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies
> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
> File "/usr/share/vdsm/clientIF.py", line 448, in
> getAllVmIoTunePolicies
> 'current_values': v.getIoTune()}
> File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTune
> result = self.getIoTuneResponse()
> File "/usr/share/vdsm/virt/vm.py", line 2816, in getIoTuneResponse
> res = self._dom.blockIoTune(
> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
> line 47, in __getattr__
> % self.vmid)
> NotConnectedError: VM u'da98112d-b9fb-4098-93fa-1f1374b41e46' was
> not started yet or was shut down
>
>
> The storage of self hosted engine multipath, pvs, lvs, seems ok...
>
> At the moment of the three nodes there is a working about 100
> virtual machines and we can't manage them.
>
> Does anyone have any ideas, what can be done то recover self
> hosted engine virtual machine?
>
> Thahk You!
> Have a nice day!
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
--
Nathanaël Blanchet
Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
blanchet(a)abes.fr
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh
--------------FB3CCABEFD28F954C51E2EC1
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p><br>
</p>
<p>anything in the journal about libvirt? (journalctl -u libvirtd)</p>
<p><br>
</p>
<p>could you share a bigger chunk of the vdsm log, demonstrating the
failed VM start?</p>
<p><br>
</p>
<p>Bests,<br>
</p>
<br>
<div class="moz-cite-prefix">On 10/30/2017 03:28 PM, Hristo Pavlov
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1509373682.39477110@f387.i.mail.ru">
<span id="result_box" class="short_text"
lang="en"><span>I tried
it already, <span id="result_box" class="short_text"
lang="en"><span>doesn't start on any of the nodes.
<br>
<br>
<br>
In a log /var/log/libvirt/qemu/HostedEngine.log at all<span
id="result_box" class="short_text"
lang="en"><span>
three nodes</span></span> <span
id="result_box"
lang="en"><span>has nothing to do with starting, as if
it had not
started.</span></span></span></span></span></span><br>
<br>
<br>
<blockquote style="border-left:1px solid #0857A6; margin:10px;
padding:0 0 0 10px;"> Понедельник, 30 октября 2017, 16:11 +02:00
от Nathanaël Blanchet <a class="moz-txt-link-rfc2396E"
href="mailto:blanchet@abes.fr"><blanchet@abes.fr></a>:<br>
<br>
<div id="">
<div class="js-helper js-readmsg-msg">
<style type="text/css"></style>
<div> <base target="_self"
href="https://e.mail.ru/">
<div id="style_15093726620000000687_BODY">
<div class="class_1509374974">
<p>Hi,</p>
<p>It happened to me too, after a live migration of
it, I shut down the hosted engine on the targeted
host, and I couldn't restart it anymore on this
specific host. But I was able to start it on the
initial one, where I initially where I deployed the
HE. It was like the lease and the libvirt host
definition staid on the first host after migration.</p>
<p>So try you may try <span
id="result_box_mailru_css_attribute_postfix"
class="short_text_mailru_css_attribute_postfix"
lang="en"><span>hosted-engine --vm-start on one
of
other hosts...<br>
</span></span></p>
<br>
<div
class="moz-cite-prefix_mailru_css_attribute_postfix">Le
30/10/2017 à 12:28, Hristo Pavlov a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:1509362918.145880938@f267.i.mail.ru"> Hi
All,<br>
<br>
Our oVirt cluster is with 3 nodes with shared fibre
channel storage, the engine virtual machine is self
hosted.<br>
<br>
Hypervisors OS: CentOS Linux release 7.3 / x86_64,
oVirt version is <span
class="st_mailru_css_attribute_postfix"><span
class="st_mailru_css_attribute_postfix">4.1.2.2.
<span
id="result_box_mailru_css_attribute_postfix"
lang="en"><span>The environment has been
working for about a year without any
problems</span></span>.<br>
<br>
After</span></span><span
id="result_box_mailru_css_attribute_postfix"
class="short_text_mailru_css_attribute_postfix"
lang="en"><span> shutdown of the hosted engine
virtual machine, it doesn't start. <br>
<br>
<span
id="result_box_mailru_css_attribute_postfix"
class="short_text_mailru_css_attribute_postfix"
lang="en"><span>Тhese commands that were
executed:<br>
<br>
hosted-engine --set-maintenance
--mode=global<br>
hosted-engine --vm-shutdown<br>
</span></span><br>
after the status of engine vm was down, we
executed start.<br>
<br>
[root@alpha] hosted-engine --vm-start<br>
VM exists and is down, destroying it<br>
Exception in thread Client localhost:54321 (most
likely raised during interpreter
shutdown):</span></span><span
class="st_mailru_css_attribute_postfix"><br>
<br>
we noticied that at vdsm.log <br>
<br>
017-10-30 13:11:04,863+0200 INFO (jsonrpc/1)
[jsonrpc.JsonRpcServer] RPC call
StorageDomain.getStats succeeded in 0.26 seconds
(__init__:533)<br>
2017-10-30 13:11:05,802+0200 INFO (jsonrpc/6)
[jsonrpc.JsonRpcServer] RPC call
Host.getAllVmStats succeeded in 0.01 seconds
(__init__:533)<br>
2017-10-30 13:11:05,825+0200 WARN (jsonrpc/2)
[virt.vm]
(vmId='da98112d-b9fb-4098-93fa-1f1374b41e46')
Failed to get metadata, domain not connected.
(vm:2765)<br>
2017-10-30 13:11:05,825+0200 ERROR (jsonrpc/2)
[jsonrpc.JsonRpcServer] Internal server error
(__init__:570)<br>
Traceback (most recent call last):<br>
File
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py",
line 565, in _handle_request<br>
res = method(**params)<br>
File
"/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py",
line 202, in _dynamicMethod<br>
result = fn(*methodArgs)<br>
File "/usr/share/vdsm/API.py", line 1454, in
getAllVmIoTunePolicies<br>
io_tune_policies_dict =
self._cif.getAllVmIoTunePolicies()<br>
File "/usr/share/vdsm/clientIF.py", line 448, in
getAllVmIoTunePolicies<br>
'current_values': v.getIoTune()}<br>
File "/usr/share/vdsm/virt/vm.py", line 2803, in
getIoTune<br>
result = self.getIoTuneResponse()<br>
File "/usr/share/vdsm/virt/vm.py", line 2816, in
getIoTuneResponse<br>
res = self._dom.blockIoTune(<br>
File
"/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
line 47, in __getattr__<br>
% self.vmid)<br>
NotConnectedError: VM
u'da98112d-b9fb-4098-93fa-1f1374b41e46' was not
started yet or was shut down<br>
<br>
<br>
The storage of self hosted engine multipath, pvs,
lvs, seems ok... <br>
<br>
<span id="result_box_mailru_css_attribute_postfix"
lang="en"><span>At the moment of the three
nodes
there is a working about 100 virtual machines
and we can't manage them.<br>
<br>
<span
id="result_box_mailru_css_attribute_postfix"
class="short_text_mailru_css_attribute_postfix"
lang="en"><span>Does
anyone have any ideas, what can be
done</span></span>
то recover self hosted engine virtual machine?<br>
<br>
Thahk You!<br>
Have a nice day!<br>
</span></span><br>
<br>
<br>
<br>
<br>
<br>
<br>
<span></span></span><br>
<br>
<br>
<br>
<br>
<fieldset
class="mimeAttachmentHeader_mailru_css_attribute_postfix"></fieldset>
<br>
<pre
wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated_mailru_css_attribute_postfix"
href="//e.mail.ru/compose/?mailto=mailto%3aUsers(a)ovirt.org"
target="_blank" rel=" noopener noreferrer"
moz-do-not-send="true">Users(a)ovirt.org</a>
<a class="moz-txt-link-freetext_mailru_css_attribute_postfix"
href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank"
rel=" noopener noreferrer"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</pre>
</blockquote>
<br>
<pre class="moz-signature_mailru_css_attribute_postfix"
cols="72">--
Nathanaël Blanchet
Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
<a class="moz-txt-link-abbreviated_mailru_css_attribute_postfix"
href="//e.mail.ru/compose/?mailto=mailto%3ablanchet(a)abes.fr"
target="_blank" rel=" noopener noreferrer"
moz-do-not-send="true">blanchet(a)abes.fr</a> </pre>
</div>
</div>
<base target="_self" href="https://e.mail.ru/">
</div>
</div>
</div>
</blockquote>
<br>
<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Francesco Romani
Senior SW Eng., Virtualization R&D
Red Hat
IRC: fromani github: @fromanirh</pre>
</body>
</html>
--------------FB3CCABEFD28F954C51E2EC1--