self hosted engine vm won't start
----ALT--D5cNXoZIQgJlULkvQvmo8ZLJGDrgBKlm1509362918 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 SGkgQWxsLAoKT3VyIG9WaXJ0IGNsdXN0ZXIgaXMgd2l0aCAzIG5vZGVzIHdpdGggc2hhcmVkIGZp YnJlIGNoYW5uZWwgc3RvcmFnZSwgdGhlIGVuZ2luZSB2aXJ0dWFsIG1hY2hpbmUgaXMgc2VsZiBo b3N0ZWQuCsKgCkh5cGVydmlzb3JzIE9TOiBDZW50T1MgTGludXggcmVsZWFzZSA3LjMgLyB4ODZf NjQsIG9WaXJ0IHZlcnNpb24gaXMgIDQuMS4yLjIuICBUaGUgZW52aXJvbm1lbnQgaGFzIGJlZW4g d29ya2luZyBmb3IgYWJvdXQgYSB5ZWFyIHdpdGhvdXQgYW55IHByb2JsZW1zIC4KCkFmdGVyIHNo dXRkb3duIG9mIHRoZSBob3N0ZWQgZW5naW5lIHZpcnR1YWwgbWFjaGluZSwgaXQgZG9lc24ndCBz dGFydC4gCgrQomhlc2UgY29tbWFuZHMgdGhhdCB3ZXJlIGV4ZWN1dGVkOgoKaG9zdGVkLWVuZ2lu ZSAtLXNldC1tYWludGVuYW5jZSAtLW1vZGU9Z2xvYmFsCmhvc3RlZC1lbmdpbmUgLS12bS1zaHV0 ZG93bgoKYWZ0ZXIgdGhlIHN0YXR1cyBvZiBlbmdpbmUgdm0gd2FzIGRvd24sIHdlIGV4ZWN1dGVk IHN0YXJ0LgoKW3Jvb3RAYWxwaGFdIGhvc3RlZC1lbmdpbmUgLS12bS1zdGFydApWTSBleGlzdHMg YW5kIGlzIGRvd24sIGRlc3Ryb3lpbmcgaXQKRXhjZXB0aW9uIGluIHRocmVhZCBDbGllbnQgbG9j YWxob3N0OjU0MzIxIChtb3N0IGxpa2VseSByYWlzZWQgZHVyaW5nIGludGVycHJldGVyIHNodXRk b3duKToKCndlIG5vdGljaWVkIHRoYXQgYXQgdmRzbS5sb2cgCgowMTctMTAtMzAgMTM6MTE6MDQs ODYzKzAyMDAgSU5GT8KgIChqc29ucnBjLzEpIFtqc29ucnBjLkpzb25ScGNTZXJ2ZXJdIFJQQyBj YWxsIFN0b3JhZ2VEb21haW4uZ2V0U3RhdHMgc3VjY2VlZGVkIGluIDAuMjYgc2Vjb25kcyAoX19p bml0X186NTMzKQoyMDE3LTEwLTMwIDEzOjExOjA1LDgwMiswMjAwIElORk/CoCAoanNvbnJwYy82 KSBbanNvbnJwYy5Kc29uUnBjU2VydmVyXSBSUEMgY2FsbCBIb3N0LmdldEFsbFZtU3RhdHMgc3Vj Y2VlZGVkIGluIDAuMDEgc2Vjb25kcyAoX19pbml0X186NTMzKQoyMDE3LTEwLTMwIDEzOjExOjA1 LDgyNSswMjAwIFdBUk7CoCAoanNvbnJwYy8yKSBbdmlydC52bV0gKHZtSWQ9J2RhOTgxMTJkLWI5 ZmItNDA5OC05M2ZhLTFmMTM3NGI0MWU0NicpIEZhaWxlZCB0byBnZXQgbWV0YWRhdGEsIGRvbWFp biBub3QgY29ubmVjdGVkLiAodm06Mjc2NSkKMjAxNy0xMC0zMCAxMzoxMTowNSw4MjUrMDIwMCBF UlJPUiAoanNvbnJwYy8yKSBbanNvbnJwYy5Kc29uUnBjU2VydmVyXSBJbnRlcm5hbCBzZXJ2ZXIg ZXJyb3IgKF9faW5pdF9fOjU3MCkKVHJhY2ViYWNrIChtb3N0IHJlY2VudCBjYWxsIGxhc3QpOgrC oCBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy95YWpzb25ycGMvX19pbml0 X18ucHkiLCBsaW5lIDU2NSwgaW4gX2hhbmRsZV9yZXF1ZXN0CsKgwqDCoCByZXMgPSBtZXRob2Qo KipwYXJhbXMpCsKgIEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20v cnBjL0JyaWRnZS5weSIsIGxpbmUgMjAyLCBpbiBfZHluYW1pY01ldGhvZArCoMKgwqAgcmVzdWx0 ID0gZm4oKm1ldGhvZEFyZ3MpCsKgIEZpbGUgIi91c3Ivc2hhcmUvdmRzbS9BUEkucHkiLCBsaW5l IDE0NTQsIGluIGdldEFsbFZtSW9UdW5lUG9saWNpZXMKwqDCoMKgIGlvX3R1bmVfcG9saWNpZXNf ZGljdCA9IHNlbGYuX2NpZi5nZXRBbGxWbUlvVHVuZVBvbGljaWVzKCkKwqAgRmlsZSAiL3Vzci9z aGFyZS92ZHNtL2NsaWVudElGLnB5IiwgbGluZSA0NDgsIGluIGdldEFsbFZtSW9UdW5lUG9saWNp ZXMKwqDCoMKgICdjdXJyZW50X3ZhbHVlcyc6IHYuZ2V0SW9UdW5lKCl9CsKgIEZpbGUgIi91c3Iv c2hhcmUvdmRzbS92aXJ0L3ZtLnB5IiwgbGluZSAyODAzLCBpbiBnZXRJb1R1bmUKwqDCoMKgIHJl c3VsdCA9IHNlbGYuZ2V0SW9UdW5lUmVzcG9uc2UoKQrCoCBGaWxlICIvdXNyL3NoYXJlL3Zkc20v dmlydC92bS5weSIsIGxpbmUgMjgxNiwgaW4gZ2V0SW9UdW5lUmVzcG9uc2UKwqDCoMKgIHJlcyA9 IHNlbGYuX2RvbS5ibG9ja0lvVHVuZSgKwqAgRmlsZSAiL3Vzci9saWIvcHl0aG9uMi43L3NpdGUt cGFja2FnZXMvdmRzbS92aXJ0L3ZpcmRvbWFpbi5weSIsIGxpbmUgNDcsIGluIF9fZ2V0YXR0cl9f CsKgwqDCoCAlIHNlbGYudm1pZCkKTm90Q29ubmVjdGVkRXJyb3I6IFZNIHUnZGE5ODExMmQtYjlm Yi00MDk4LTkzZmEtMWYxMzc0YjQxZTQ2JyB3YXMgbm90IHN0YXJ0ZWQgeWV0IG9yIHdhcyBzaHV0 IGRvd24KCgpUaGUgc3RvcmFnZSBvZiBzZWxmIGhvc3RlZCBlbmdpbmUgbXVsdGlwYXRoLCBwdnMs IGx2cywgc2VlbXMgb2suLi4gCgpBdCB0aGUgbW9tZW50IG9mIHRoZSB0aHJlZSBub2RlcyB0aGVy ZSBpcyBhIHdvcmtpbmcgYWJvdXQgMTAwIHZpcnR1YWwgbWFjaGluZXMgYW5kIHdlIGNhbid0IG1h bmFnZSB0aGVtLgoKRG9lcyBhbnlvbmUgaGF2ZSBhbnkgaWRlYXMsIHdoYXQgY2FuIGJlIGRvbmUg 0YLQviByZWNvdmVyIHNlbGYgaG9zdGVkIGVuZ2luZSB2aXJ0dWFsIG1hY2hpbmU/CgpUaGFoayBZ b3UhCkhhdmUgYSBuaWNlIGRheSEKCgoKCgoKCu+7vwoKCgo= ----ALT--D5cNXoZIQgJlULkvQvmo8ZLJGDrgBKlm1509362918 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 CjxIVE1MPjxCT0RZPkhpIEFsbCw8YnI+PGJyPk91ciBvVmlydCBjbHVzdGVyIGlzIHdpdGggMyBu b2RlcyB3aXRoIHNoYXJlZCBmaWJyZSBjaGFubmVsIHN0b3JhZ2UsIHRoZSBlbmdpbmUgdmlydHVh bCBtYWNoaW5lIGlzIHNlbGYgaG9zdGVkLjxicj4mbmJzcDs8YnI+SHlwZXJ2aXNvcnMgT1M6IENl bnRPUyBMaW51eCByZWxlYXNlIDcuMyAvIHg4Nl82NCwgb1ZpcnQgdmVyc2lvbiBpcyA8c3BhbiBj bGFzcz0ic3QiPjxzcGFuIGNsYXNzPSJzdCI+NC4xLjIuMi4gPHNwYW4gaWQ9InJlc3VsdF9ib3gi IGxhbmc9ImVuIj48c3Bhbj5UaGUgZW52aXJvbm1lbnQgaGFzIGJlZW4gd29ya2luZyBmb3IgYWJv dXQgYSB5ZWFyIHdpdGhvdXQgYW55IHByb2JsZW1zPC9zcGFuPjwvc3Bhbj4uPGJyPjxicj5BZnRl cjwvc3Bhbj48L3NwYW4+PHNwYW4gaWQ9InJlc3VsdF9ib3giIGNsYXNzPSJzaG9ydF90ZXh0IiBs YW5nPSJlbiI+PHNwYW4+IHNodXRkb3duIG9mIHRoZSBob3N0ZWQgZW5naW5lIHZpcnR1YWwgbWFj aGluZSwgaXQgZG9lc24ndCBzdGFydC4gPGJyPjxicj48c3BhbiBpZD0icmVzdWx0X2JveCIgY2xh c3M9InNob3J0X3RleHQiIGxhbmc9ImVuIj48c3Bhbj7Qomhlc2UgY29tbWFuZHMgdGhhdCB3ZXJl IGV4ZWN1dGVkOjxicj48YnI+aG9zdGVkLWVuZ2luZSAtLXNldC1tYWludGVuYW5jZSAtLW1vZGU9 Z2xvYmFsPGJyPmhvc3RlZC1lbmdpbmUgLS12bS1zaHV0ZG93bjxicj4gPC9zcGFuPjwvc3Bhbj48 YnI+YWZ0ZXIgdGhlIHN0YXR1cyBvZiBlbmdpbmUgdm0gd2FzIGRvd24sIHdlIGV4ZWN1dGVkIHN0 YXJ0Ljxicj48YnI+W3Jvb3RAYWxwaGFdIGhvc3RlZC1lbmdpbmUgLS12bS1zdGFydDxicj5WTSBl eGlzdHMgYW5kIGlzIGRvd24sIGRlc3Ryb3lpbmcgaXQ8YnI+RXhjZXB0aW9uIGluIHRocmVhZCBD bGllbnQgbG9jYWxob3N0OjU0MzIxIChtb3N0IGxpa2VseSByYWlzZWQgZHVyaW5nIGludGVycHJl dGVyIHNodXRkb3duKTo8L3NwYW4+PC9zcGFuPjxzcGFuIGNsYXNzPSJzdCI+PGJyPjxicj53ZSBu b3RpY2llZCB0aGF0IGF0IHZkc20ubG9nIDxicj48YnI+MDE3LTEwLTMwIDEzOjExOjA0LDg2Mysw MjAwIElORk8mbmJzcDsgKGpzb25ycGMvMSkgW2pzb25ycGMuSnNvblJwY1NlcnZlcl0gUlBDIGNh bGwgU3RvcmFnZURvbWFpbi5nZXRTdGF0cyBzdWNjZWVkZWQgaW4gMC4yNiBzZWNvbmRzIChfX2lu aXRfXzo1MzMpPGJyPjIwMTctMTAtMzAgMTM6MTE6MDUsODAyKzAyMDAgSU5GTyZuYnNwOyAoanNv bnJwYy82KSBbanNvbnJwYy5Kc29uUnBjU2VydmVyXSBSUEMgY2FsbCBIb3N0LmdldEFsbFZtU3Rh dHMgc3VjY2VlZGVkIGluIDAuMDEgc2Vjb25kcyAoX19pbml0X186NTMzKTxicj4yMDE3LTEwLTMw IDEzOjExOjA1LDgyNSswMjAwIFdBUk4mbmJzcDsgKGpzb25ycGMvMikgW3ZpcnQudm1dICh2bUlk PSdkYTk4MTEyZC1iOWZiLTQwOTgtOTNmYS0xZjEzNzRiNDFlNDYnKSBGYWlsZWQgdG8gZ2V0IG1l dGFkYXRhLCBkb21haW4gbm90IGNvbm5lY3RlZC4gKHZtOjI3NjUpPGJyPjIwMTctMTAtMzAgMTM6 MTE6MDUsODI1KzAyMDAgRVJST1IgKGpzb25ycGMvMikgW2pzb25ycGMuSnNvblJwY1NlcnZlcl0g SW50ZXJuYWwgc2VydmVyIGVycm9yIChfX2luaXRfXzo1NzApPGJyPlRyYWNlYmFjayAobW9zdCBy ZWNlbnQgY2FsbCBsYXN0KTo8YnI+Jm5ic3A7IEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRl LXBhY2thZ2VzL3lhanNvbnJwYy9fX2luaXRfXy5weSIsIGxpbmUgNTY1LCBpbiBfaGFuZGxlX3Jl cXVlc3Q8YnI+Jm5ic3A7Jm5ic3A7Jm5ic3A7IHJlcyA9IG1ldGhvZCgqKnBhcmFtcyk8YnI+Jm5i c3A7IEZpbGUgIi91c3IvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3Zkc20vcnBjL0JyaWRn ZS5weSIsIGxpbmUgMjAyLCBpbiBfZHluYW1pY01ldGhvZDxicj4mbmJzcDsmbmJzcDsmbmJzcDsg cmVzdWx0ID0gZm4oKm1ldGhvZEFyZ3MpPGJyPiZuYnNwOyBGaWxlICIvdXNyL3NoYXJlL3Zkc20v QVBJLnB5IiwgbGluZSAxNDU0LCBpbiBnZXRBbGxWbUlvVHVuZVBvbGljaWVzPGJyPiZuYnNwOyZu YnNwOyZuYnNwOyBpb190dW5lX3BvbGljaWVzX2RpY3QgPSBzZWxmLl9jaWYuZ2V0QWxsVm1Jb1R1 bmVQb2xpY2llcygpPGJyPiZuYnNwOyBGaWxlICIvdXNyL3NoYXJlL3Zkc20vY2xpZW50SUYucHki LCBsaW5lIDQ0OCwgaW4gZ2V0QWxsVm1Jb1R1bmVQb2xpY2llczxicj4mbmJzcDsmbmJzcDsmbmJz cDsgJ2N1cnJlbnRfdmFsdWVzJzogdi5nZXRJb1R1bmUoKX08YnI+Jm5ic3A7IEZpbGUgIi91c3Iv c2hhcmUvdmRzbS92aXJ0L3ZtLnB5IiwgbGluZSAyODAzLCBpbiBnZXRJb1R1bmU8YnI+Jm5ic3A7 Jm5ic3A7Jm5ic3A7IHJlc3VsdCA9IHNlbGYuZ2V0SW9UdW5lUmVzcG9uc2UoKTxicj4mbmJzcDsg RmlsZSAiL3Vzci9zaGFyZS92ZHNtL3ZpcnQvdm0ucHkiLCBsaW5lIDI4MTYsIGluIGdldElvVHVu ZVJlc3BvbnNlPGJyPiZuYnNwOyZuYnNwOyZuYnNwOyByZXMgPSBzZWxmLl9kb20uYmxvY2tJb1R1 bmUoPGJyPiZuYnNwOyBGaWxlICIvdXNyL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy92ZHNt L3ZpcnQvdmlyZG9tYWluLnB5IiwgbGluZSA0NywgaW4gX19nZXRhdHRyX188YnI+Jm5ic3A7Jm5i c3A7Jm5ic3A7ICUgc2VsZi52bWlkKTxicj5Ob3RDb25uZWN0ZWRFcnJvcjogVk0gdSdkYTk4MTEy ZC1iOWZiLTQwOTgtOTNmYS0xZjEzNzRiNDFlNDYnIHdhcyBub3Qgc3RhcnRlZCB5ZXQgb3Igd2Fz IHNodXQgZG93bjxicj48YnI+PGJyPlRoZSBzdG9yYWdlIG9mIHNlbGYgaG9zdGVkIGVuZ2luZSBt dWx0aXBhdGgsIHB2cywgbHZzLCBzZWVtcyBvay4uLiA8YnI+PGJyPjxzcGFuIGlkPSJyZXN1bHRf Ym94IiBsYW5nPSJlbiI+PHNwYW4+QXQgdGhlIG1vbWVudCBvZiB0aGUgdGhyZWUgbm9kZXMgdGhl cmUgaXMgYSB3b3JraW5nIGFib3V0IDEwMCB2aXJ0dWFsIG1hY2hpbmVzIGFuZCB3ZSBjYW4ndCBt YW5hZ2UgdGhlbS48YnI+PGJyPjxzcGFuIGlkPSJyZXN1bHRfYm94IiBjbGFzcz0ic2hvcnRfdGV4 dCIgbGFuZz0iZW4iPjxzcGFuPkRvZXMgYW55b25lIGhhdmUgYW55IGlkZWFzLCB3aGF0IGNhbiBi ZSBkb25lPC9zcGFuPjwvc3Bhbj4g0YLQviByZWNvdmVyIHNlbGYgaG9zdGVkIGVuZ2luZSB2aXJ0 dWFsIG1hY2hpbmU/PGJyPjxicj5UaGFoayBZb3UhPGJyPkhhdmUgYSBuaWNlIGRheSE8YnI+PC9z cGFuPjwvc3Bhbj48YnI+PGJyPjxicj48YnI+PGJyPjxicj48YnI+PHNwYW4+77u/PC9zcGFuPjwv c3Bhbj48YnI+PGJyPjxicj48YnI+PC9CT0RZPjwvSFRNTD4K ----ALT--D5cNXoZIQgJlULkvQvmo8ZLJGDrgBKlm1509362918--
Hi, you will probably need to check libvirt and qemu logs to see why the domain crashed. My colleagues from the virt team will be probably able to point you to more exact places. Best regards Martin Sivak On Mon, Oct 30, 2017 at 12:28 PM, Hristo Pavlov <ichkata_66@mail.ru> wrote:
Hi All,
Our oVirt cluster is with 3 nodes with shared fibre channel storage, the engine virtual machine is self hosted.
Hypervisors OS: CentOS Linux release 7.3 / x86_64, oVirt version is 4.1.2.2. The environment has been working for about a year without any problems.
After shutdown of the hosted engine virtual machine, it doesn't start.
Тhese commands that were executed:
hosted-engine --set-maintenance --mode=global hosted-engine --vm-shutdown
after the status of engine vm was down, we executed start.
[root@alpha] hosted-engine --vm-start VM exists and is down, destroying it Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown):
we noticied that at vdsm.log
017-10-30 13:11:04,863+0200 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call StorageDomain.getStats succeeded in 0.26 seconds (__init__:533) 2017-10-30 13:11:05,802+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:533) 2017-10-30 13:11:05,825+0200 WARN (jsonrpc/2) [virt.vm] (vmId='da98112d-b9fb-4098-93fa-1f1374b41e46') Failed to get metadata, domain not connected. (vm:2765) 2017-10-30 13:11:05,825+0200 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:570) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 565, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1454, in getAllVmIoTunePolicies io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() File "/usr/share/vdsm/clientIF.py", line 448, in getAllVmIoTunePolicies 'current_values': v.getIoTune()} File "/usr/share/vdsm/virt/vm.py", line 2803, in getIoTune result = self.getIoTuneResponse() File "/usr/share/vdsm/virt/vm.py", line 2816, in getIoTuneResponse res = self._dom.blockIoTune( File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__ % self.vmid) NotConnectedError: VM u'da98112d-b9fb-4098-93fa-1f1374b41e46' was not started yet or was shut down
The storage of self hosted engine multipath, pvs, lvs, seems ok...
At the moment of the three nodes there is a working about 100 virtual machines and we can't manage them.
Does anyone have any ideas, what can be done то recover self hosted engine virtual machine?
Thahk You! Have a nice day!
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Hristo Pavlov -
Martin Sivak