[ovirt-users] self hosted engine vm won't start

[Hristo Pavlov]

Hi All,

Our oVirt cluster is with 3 nodes with shared fibre channel storage, the engine virtual machine is self hosted.
 
Hypervisors OS: CentOS Linux release 7.3 / x86_64, oVirt version is  4.1.2.2.  The environment has been working for about a year without any problems .

After shutdown of the hosted engine virtual machine, it doesn't start. 

Тhese commands that were executed:

hosted-engine --set-maintenance --mode=global
hosted-engine --vm-shutdown

after the status of engine vm was down, we executed start.

[root at alpha] hosted-engine --vm-start
VM exists and is down, destroying it
Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown):

we noticied that at vdsm.log 

017-10-30 13:11:04,863+0200 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats succeeded in 0.26 seconds (__init__:533)
2017-10-30 13:11:05,802+0200 INFO  (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:533)
2017-10-30 13:11:05,825+0200 WARN  (jsonrpc/2) [virt.vm] (vmId='da98112d-b9fb-4098-93fa-1f1374b41e46') Failed to get metadata, domain not connected. (vm:2765)
2017-10-30 13:11:05,825+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:570)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 565, in _handle_request
    res = method(**params)
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod
    result = fn(*methodArgs)
  File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies
    io_tune_policies_dict = self._cif.getAllVmIoTunePolicies()
  File "/usr/share/vdsm/clientIF.py", line 448, in getAllVmIoTunePolicies
    'current_values': v.getIoTune()}
  File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTune
    result = self.getIoTuneResponse()
  File "/usr/share/vdsm/virt/vm.py", line 2816, in getIoTuneResponse
    res = self._dom.blockIoTune(
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__
    % self.vmid)
NotConnectedError: VM u'da98112d-b9fb-4098-93fa-1f1374b41e46' was not started yet or was shut down


The storage of self hosted engine multipath, pvs, lvs, seems ok... 

At the moment of the three nodes there is a working about 100 virtual machines and we can't manage them.

Does anyone have any ideas, what can be done то recover self hosted engine virtual machine?

Thahk You!
Have a nice day!











-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20171030/32a262ee/attachment.html>