Fwd: Re: self hosted engine vm won't start

30 Oct 2017

      ...
Hi All,
Our oVirt cluster is with 3 nodes with shared fibre channel storage,=20
the engine virtual machine is self hosted.
Hypervisors OS: CentOS Linux release 7.3 / x86_64, oVirt version is=20
4.1.2.2. The environment has been working for about a year without any=20
problems.
Aftershutdown of the hosted engine virtual machine, it doesn't start.
=D0=A2hese commands that were executed:
hosted-engine --set-maintenance --mode=3Dglobal
hosted-engine --vm-shutdown
after the status of engine vm was down, we executed start.
[root@alpha] hosted-engine --vm-start
VM exists and is down, destroying it
Exception in thread Client localhost:54321 (most likely raised during=20
interpreter shutdown):
we noticied that at vdsm.log
017-10-30 13:11:04,863+0200 INFO=C2=A0 (jsonrpc/1) [jsonrpc.JsonRpcServ=
er]=20
RPC call StorageDomain.getStats succeeded in 0.26 seconds (__init__:533=
)
2017-10-30 13:11:05,802+0200 INFO=C2=A0 (jsonrpc/6) [jsonrpc.JsonRpcSer=
ver]=20
RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:533)
2017-10-30 13:11:05,825+0200 WARN=C2=A0 (jsonrpc/2) [virt.vm]=20
(vmId=3D'da98112d-b9fb-4098-93fa-1f1374b41e46') Failed to get metadata,=
=20
domain not connected. (vm:2765)
2017-10-30 13:11:05,825+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer]=20
Internal server error (__init__:570)
Traceback (most recent call last):
=C2=A0 File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", l=
ine=20
565, in _handle_request
=C2=A0=C2=A0=C2=A0 res =3D method(**params)
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line=
=20
202, in _dynamicMethod
=C2=A0=C2=A0=C2=A0 result =3D fn(*methodArgs)
=C2=A0 File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolic=
ies
=C2=A0=C2=A0=C2=A0 io_tune_policies_dict =3D self._cif.getAllVmIoTunePo=
...
=C2=A0 File "/usr/share/vdsm/clientIF.py", line 448, in getAllVmIoTuneP=
...
=C2=A0=C2=A0=C2=A0 'current_values': v.getIoTune()}
=C2=A0 File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTune
=C2=A0=C2=A0=C2=A0 result =3D self.getIoTuneResponse()
=C2=A0 File "/usr/share/vdsm/virt/vm.py", line 2816, in getIoTuneRespon=
se
=C2=A0=C2=A0=C2=A0 res =3D self._dom.blockIoTune(
=C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", =
This is a multi-part message in MIME format.
--------------134568D2CB5386CDC99CD65A
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable

Hi,

It happened to me too, after a live migration of it, I shut down the=20
hosted engine on the targeted host, and I couldn't restart it anymore on=20
this specific host. But I was able to start it on the initial one, where=20
I initially where I deployed the HE. It was like the lease and the=20
libvirt host definition staid on the first host after migration.

So try you may try hosted-engine --vm-start on one of other hosts...

Le 30/10/2017 =C3=A0 12:28, Hristo Pavlov a =C3=A9crit=C2=A0:
licies()
olicies
line=20
...
47, in __getattr__
=C2=A0=C2=A0=C2=A0 % self.vmid)
NotConnectedError: VM u'da98112d-b9fb-4098-93fa-1f1374b41e46' was not=20
started yet or was shut down
The storage of self hosted engine multipath, pvs, lvs, seems ok...
At the moment of the three nodes there is a working about 100 virtual=20
machines and we can't manage them.
Does anyone have any ideas, what can be done =D1=82=D0=BE recover self =
hosted=20
engine virtual machine?
Thahk You!
Have a nice day!
=EF=BB=BF
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--=20
Nathana=C3=ABl Blanchet

Supervision r=C3=A9seau
P=C3=B4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=C3=A9l. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
blanchet@abes.fr =20

--------------134568D2CB5386CDC99CD65A
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html>
  <head>

    <meta http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf=
-8">
  </head>
  <body text=3D"#000000" bgcolor=3D"#FFFFFF">
    <p> </p>
    <div class=3D"moz-forward-container">
      <p>Hi,</p>
      <p>It happened to me too, after a live migration of it, I shut
        down the hosted engine on the targeted host, and I couldn't
        restart it anymore on this specific host. But I was able to
        start it on the initial one, where I initially where I deployed
        the HE. It was like the lease and the libvirt host definition
        staid on the first host after migration.</p>
      <p>So try you may try <span id=3D"result_box" class=3D"short_text"
          lang=3D"en"><span>hosted-engine --vm-start on one of other
            hosts...<br>
          </span></span></p>
      <br>
      <div class=3D"moz-cite-prefix">Le 30/10/2017 =C3=A0 12:28, Hristo P=
avlov
        a =C3=A9crit=C2=A0:<br>
      </div>
      <blockquote type=3D"cite"
        cite=3D"mid:1509362918.145880938@f267.i.mail.ru"> Hi All,<br>
        <br>
        Our oVirt cluster is with 3 nodes with shared fibre channel
        storage, the engine virtual machine is self hosted.<br>
        =C2=A0<br>
        Hypervisors OS: CentOS Linux release 7.3 / x86_64, oVirt version
        is <span class=3D"st"><span class=3D"st">4.1.2.2. <span
              id=3D"result_box" lang=3D"en"><span>The environment has bee=
n
                working for about a year without any problems</span></spa=
n>.<br>
            <br>
            After</span></span><span id=3D"result_box" class=3D"short_tex=
t"
          lang=3D"en"><span> shutdown of the hosted engine virtual
            machine, it doesn't start. <br>
            <br>
            <span id=3D"result_box" class=3D"short_text" lang=3D"en"><spa=
n>=D0=A2hese
                commands that were executed:<br>
                <br>
                hosted-engine --set-maintenance --mode=3Dglobal<br>
                hosted-engine --vm-shutdown<br>
              </span></span><br>
            after the status of engine vm was down, we executed start.<br=
...
<br>
            [root@alpha] hosted-engine --vm-start<br>
            VM exists and is down, destroying it<br>
            Exception in thread Client localhost:54321 (most likely
            raised during interpreter shutdown):</span></span><span
          class=3D"st"><br>
          <br>
          we noticied that at vdsm.log <br>
          <br>
          017-10-30 13:11:04,863+0200 INFO=C2=A0 (jsonrpc/1)
          [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats
          succeeded in 0.26 seconds (__init__:533)<br>
          2017-10-30 13:11:05,802+0200 INFO=C2=A0 (jsonrpc/6)
          [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded
          in 0.01 seconds (__init__:533)<br>
          2017-10-30 13:11:05,825+0200 WARN=C2=A0 (jsonrpc/2) [virt.vm]
          (vmId=3D'da98112d-b9fb-4098-93fa-1f1374b41e46') Failed to get
          metadata, domain not connected. (vm:2765)<br>
          2017-10-30 13:11:05,825+0200 ERROR (jsonrpc/2)
          [jsonrpc.JsonRpcServer] Internal server error (__init__:570)<br=
...
Traceback (most recent call last):<br>
          =C2=A0 File
          "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
          565, in _handle_request<br>
          =C2=A0=C2=A0=C2=A0 res =3D method(**params)<br>
          =C2=A0 File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.p=
y",
          line 202, in _dynamicMethod<br>
          =C2=A0=C2=A0=C2=A0 result =3D fn(*methodArgs)<br>
          =C2=A0 File "/usr/share/vdsm/API.py", line 1454, in
          getAllVmIoTunePolicies<br>
          =C2=A0=C2=A0=C2=A0 io_tune_policies_dict =3D self._cif.getAllVm=
IoTunePolicies()<br>
          =C2=A0 File "/usr/share/vdsm/clientIF.py", line 448, in
          getAllVmIoTunePolicies<br>
          =C2=A0=C2=A0=C2=A0 'current_values': v.getIoTune()}<br>
          =C2=A0 File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTu=
ne<br>
          =C2=A0=C2=A0=C2=A0 result =3D self.getIoTuneResponse()<br>
          =C2=A0 File "/usr/share/vdsm/virt/vm.py", line 2816, in
          getIoTuneResponse<br>
          =C2=A0=C2=A0=C2=A0 res =3D self._dom.blockIoTune(<br>
          =C2=A0 File
          "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
          line 47, in __getattr__<br>
          =C2=A0=C2=A0=C2=A0 % self.vmid)<br>
          NotConnectedError: VM u'da98112d-b9fb-4098-93fa-1f1374b41e46'
          was not started yet or was shut down<br>
          <br>
          <br>
          The storage of self hosted engine multipath, pvs, lvs, seems
          ok... <br>
          <br>
          <span id=3D"result_box" lang=3D"en"><span>At the moment of the
              three nodes there is a working about 100 virtual machines
              and we can't manage them.<br>
              <br>
              <span id=3D"result_box" class=3D"short_text" lang=3D"en"><s=
pan>Does
                  anyone have any ideas, what can be done</span></span>
              =D1=82=D0=BE recover self hosted engine virtual machine?<br=
...
<br>
              Thahk You!<br>
              Have a nice day!<br>
            </span></span><br>
          <br>
          <br>
          <br>
          <br>
          <br>
          <br>
          <span>=EF=BB=BF</span></span><br>
        <br>
        <br>
        <br>
        <br>
        <fieldset class=3D"mimeAttachmentHeader"></fieldset>
        <br>
        <pre wrap=3D"">_______________________________________________
Users mailing list
<a class=3D"moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org" moz=
-do-not-send=3D"true">Users@ovirt.org</a>
<a class=3D"moz-txt-link-freetext" href=3D"http://lists.ovirt.org/mailman=
/listinfo/users" moz-do-not-send=3D"true">http://lists.ovirt.org/mailman/=
listinfo/users</a>
</pre>
      </blockquote>
      <br>
      <pre class=3D"moz-signature" cols=3D"72">--=20
Nathana=C3=ABl Blanchet

Supervision r=C3=A9seau
P=C3=B4le Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5 =09
T=C3=A9l. 33 (0)4 67 54 84 55
Fax  33 (0)4 67 54 84 14
<a class=3D"moz-txt-link-abbreviated" href=3D"mailto:blanchet@abes.fr" mo=
z-do-not-send=3D"true">blanchet@abes.fr</a> </pre>
    </div>
  </body>
</html>

--------------134568D2CB5386CDC99CD65A--

Nathanaël Blanchet

Hristo Pavlov

tags

participants (2)