This is a multi-part message in MIME format. --------------FB3CCABEFD28F954C51E2EC1 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Hi, anything in the journal about libvirt? (journalctl -u libvirtd) could you share a bigger chunk of the vdsm log, demonstrating the failed VM start? Bests, On 10/30/2017 03:28 PM, Hristo Pavlov wrote:
I tried it already, doesn't start on any of the nodes.
In a log /var/log/libvirt/qemu/HostedEngine.log at allthree nodes has nothing to do with starting, as if it had not started.
Понедельник, 30 октября 2017, 16:11 +02:00 от Nathanaël Blanchet <blanchet@abes.fr>:
Hi,
It happened to me too, after a live migration of it, I shut down the hosted engine on the targeted host, and I couldn't restart it anymore on this specific host. But I was able to start it on the initial one, where I initially where I deployed the HE. It was like the lease and the libvirt host definition staid on the first host after migration.
So try you may try hosted-engine --vm-start on one of other hosts...
Le 30/10/2017 à 12:28, Hristo Pavlov a écrit :
Hi All,
Our oVirt cluster is with 3 nodes with shared fibre channel storage, the engine virtual machine is self hosted. Hypervisors OS: CentOS Linux release 7.3 / x86_64, oVirt version is 4.1.2.2. The environment has been working for about a year without any problems.
Aftershutdown of the hosted engine virtual machine, it doesn't start.
Тhese commands that were executed:
hosted-engine --set-maintenance --mode=global hosted-engine --vm-shutdown
after the status of engine vm was down, we executed start.
[root@alpha] hosted-engine --vm-start VM exists and is down, destroying it Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown):
we noticied that at vdsm.log
017-10-30 13:11:04,863+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats succeeded in 0.26 seconds (__init__:533) 2017-10-30 13:11:05,802+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:533) 2017-10-30 13:11:05,825+0200 WARN (jsonrpc/2) [virt.vm] (vmId='da98112d-b9fb-4098-93fa-1f1374b41e46') Failed to get metadata, domain not connected. (vm:2765) 2017-10-30 13:11:05,825+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:570) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 565, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() File "/usr/share/vdsm/clientIF.py", line 448, in getAllVmIoTunePolicies 'current_values': v.getIoTune()} File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTune result = self.getIoTuneResponse() File "/usr/share/vdsm/virt/vm.py", line 2816, in getIoTuneResponse res = self._dom.blockIoTune( File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__ % self.vmid) NotConnectedError: VM u'da98112d-b9fb-4098-93fa-1f1374b41e46' was not started yet or was shut down
The storage of self hosted engine multipath, pvs, lvs, seems ok...
At the moment of the three nodes there is a working about 100 virtual machines and we can't manage them.
Does anyone have any ideas, what can be done то recover self hosted engine virtual machine?
Thahk You! Have a nice day!
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Nathanaël Blanchet
Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet@abes.fr
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh --------------FB3CCABEFD28F954C51E2EC1 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Hi,</p> <p><br> </p> <p>anything in the journal about libvirt? (journalctl -u libvirtd)</p> <p><br> </p> <p>could you share a bigger chunk of the vdsm log, demonstrating the failed VM start?</p> <p><br> </p> <p>Bests,<br> </p> <br> <div class="moz-cite-prefix">On 10/30/2017 03:28 PM, Hristo Pavlov wrote:<br> </div> <blockquote type="cite" cite="mid:1509373682.39477110@f387.i.mail.ru"> <span id="result_box" class="short_text" lang="en"><span>I tried it already, <span id="result_box" class="short_text" lang="en"><span>doesn't start on any of the nodes. <br> <br> <br> In a log /var/log/libvirt/qemu/HostedEngine.log at all<span id="result_box" class="short_text" lang="en"><span> three nodes</span></span> <span id="result_box" lang="en"><span>has nothing to do with starting, as if it had not started.</span></span></span></span></span></span><br> <br> <br> <blockquote style="border-left:1px solid #0857A6; margin:10px; padding:0 0 0 10px;"> Понедельник, 30 октября 2017, 16:11 +02:00 от Nathanaël Blanchet <a class="moz-txt-link-rfc2396E" href="mailto:blanchet@abes.fr"><blanchet@abes.fr></a>:<br> <br> <div id=""> <div class="js-helper js-readmsg-msg"> <style type="text/css"></style> <div> <base target="_self" href="https://e.mail.ru/"> <div id="style_15093726620000000687_BODY"> <div class="class_1509374974"> <p>Hi,</p> <p>It happened to me too, after a live migration of it, I shut down the hosted engine on the targeted host, and I couldn't restart it anymore on this specific host. But I was able to start it on the initial one, where I initially where I deployed the HE. It was like the lease and the libvirt host definition staid on the first host after migration.</p> <p>So try you may try <span id="result_box_mailru_css_attribute_postfix" class="short_text_mailru_css_attribute_postfix" lang="en"><span>hosted-engine --vm-start on one of other hosts...<br> </span></span></p> <br> <div class="moz-cite-prefix_mailru_css_attribute_postfix">Le 30/10/2017 à 12:28, Hristo Pavlov a écrit :<br> </div> <blockquote type="cite" cite="mid:1509362918.145880938@f267.i.mail.ru"> Hi All,<br> <br> Our oVirt cluster is with 3 nodes with shared fibre channel storage, the engine virtual machine is self hosted.<br> <br> Hypervisors OS: CentOS Linux release 7.3 / x86_64, oVirt version is <span class="st_mailru_css_attribute_postfix"><span class="st_mailru_css_attribute_postfix">4.1.2.2. <span id="result_box_mailru_css_attribute_postfix" lang="en"><span>The environment has been working for about a year without any problems</span></span>.<br> <br> After</span></span><span id="result_box_mailru_css_attribute_postfix" class="short_text_mailru_css_attribute_postfix" lang="en"><span> shutdown of the hosted engine virtual machine, it doesn't start. <br> <br> <span id="result_box_mailru_css_attribute_postfix" class="short_text_mailru_css_attribute_postfix" lang="en"><span>Тhese commands that were executed:<br> <br> hosted-engine --set-maintenance --mode=global<br> hosted-engine --vm-shutdown<br> </span></span><br> after the status of engine vm was down, we executed start.<br> <br> [root@alpha] hosted-engine --vm-start<br> VM exists and is down, destroying it<br> Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown):</span></span><span class="st_mailru_css_attribute_postfix"><br> <br> we noticied that at vdsm.log <br> <br> 017-10-30 13:11:04,863+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats succeeded in 0.26 seconds (__init__:533)<br> 2017-10-30 13:11:05,802+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:533)<br> 2017-10-30 13:11:05,825+0200 WARN (jsonrpc/2) [virt.vm] (vmId='da98112d-b9fb-4098-93fa-1f1374b41e46') Failed to get metadata, domain not connected. (vm:2765)<br> 2017-10-30 13:11:05,825+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:570)<br> Traceback (most recent call last):<br> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 565, in _handle_request<br> res = method(**params)<br> File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod<br> result = fn(*methodArgs)<br> File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies<br> io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()<br> File "/usr/share/vdsm/clientIF.py", line 448, in getAllVmIoTunePolicies<br> 'current_values': v.getIoTune()}<br> File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTune<br> result = self.getIoTuneResponse()<br> File "/usr/share/vdsm/virt/vm.py", line 2816, in getIoTuneResponse<br> res = self._dom.blockIoTune(<br> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__<br> % self.vmid)<br> NotConnectedError: VM u'da98112d-b9fb-4098-93fa-1f1374b41e46' was not started yet or was shut down<br> <br> <br> The storage of self hosted engine multipath, pvs, lvs, seems ok... <br> <br> <span id="result_box_mailru_css_attribute_postfix" lang="en"><span>At the moment of the three nodes there is a working about 100 virtual machines and we can't manage them.<br> <br> <span id="result_box_mailru_css_attribute_postfix" class="short_text_mailru_css_attribute_postfix" lang="en"><span>Does anyone have any ideas, what can be done</span></span> то recover self hosted engine virtual machine?<br> <br> Thahk You!<br> Have a nice day!<br> </span></span><br> <br> <br> <br> <br> <br> <br> <span></span></span><br> <br> <br> <br> <br> <fieldset class="mimeAttachmentHeader_mailru_css_attribute_postfix"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated_mailru_css_attribute_postfix" href="//e.mail.ru/compose/?mailto=mailto%3aUsers@ovirt.org" target="_blank" rel=" noopener noreferrer" moz-do-not-send="true">Users@ovirt.org</a> <a class="moz-txt-link-freetext_mailru_css_attribute_postfix" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank" rel=" noopener noreferrer" moz-do-not-send="true">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <pre class="moz-signature_mailru_css_attribute_postfix" cols="72">-- Nathanaël Blanchet Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a class="moz-txt-link-abbreviated_mailru_css_attribute_postfix" href="//e.mail.ru/compose/?mailto=mailto%3ablanchet@abes.fr" target="_blank" rel=" noopener noreferrer" moz-do-not-send="true">blanchet@abes.fr</a> </pre> </div> </div> <base target="_self" href="https://e.mail.ru/"> </div> </div> </div> </blockquote> <br> <br> <br> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <pre class="moz-signature" cols="72">-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh</pre> </body> </html> --------------FB3CCABEFD28F954C51E2EC1--