<div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div>Hello.<br></div>I&#39;m on 4.1.3 with self hosted engine and glusterfs as storage.<br></div>I updated the kernel  on engine so I executed these steps:<br><br></div>- enable global maintenace from the web admin gui<br></div>- wait some minutes<br></div>- shutdown the engine vm from inside its OS<br></div>- wait some minutes<br></div>- execute on one host<br></div>[root@ovirt02 ~]# hosted-engine --set-maintenance --mode=none<br><br></div>I see that the qemu-kvm process for the engine starts on two hosts and then on one of them it gets a &quot;kill -15&quot; and stops<br></div>Is it expected behaviour? It seems somehow dangerous to me..<br><br>- when in maintenance<br><br>[root@ovirt02 ~]# hosted-engine --vm-status<br><br><br>!! Cluster is in GLOBAL MAINTENANCE mode !!<br><br></div><div><br>--== Host 1 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt01.localdomain.local<br>Host ID                            : 1<br>Engine status                      : {&quot;health&quot;: &quot;good&quot;, &quot;vm&quot;: &quot;up&quot;, &quot;detail&quot;: &quot;up&quot;}<br>Score                              : 2597<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 7931c5c3<br>local_conf_timestamp               : 19811<br>Host timestamp                     : 19794<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=19794 (Sun Jul  9 21:31:50 2017)<br>    host-id=1<br>    score=2597<br>    vm_conf_refresh_time=19811 (Sun Jul  9 21:32:06 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=GlobalMaintenance<br>    stopped=False<br><br><br>--== Host 2 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : 192.168.150.103<br>Host ID                            : 2<br>Engine status                      : {&quot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;unknown&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 616ceb02<br>local_conf_timestamp               : 2829<br>Host timestamp                     : 2812<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=2812 (Sun Jul  9 21:31:52 2017)<br>    host-id=2<br>    score=3400<br>    vm_conf_refresh_time=2829 (Sun Jul  9 21:32:09 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=GlobalMaintenance<br>    stopped=False<br><br><br>--== Host 3 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt03.localdomain.local<br>Host ID                            : 3<br>Engine status                      : {&quot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;unknown&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 871204b2<br>local_conf_timestamp               : 24584<br>Host timestamp                     : 24567<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=24567 (Sun Jul  9 21:31:52 2017)<br>    host-id=3<br>    score=3400<br>    vm_conf_refresh_time=24584 (Sun Jul  9 21:32:09 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=GlobalMaintenance<br>    stopped=False<br><br><br>!! Cluster is in GLOBAL MAINTENANCE mode !!<br>[root@ovirt02 ~]#<br><br><br></div><div>- then I exit global maintenance<br>[root@ovirt02 ~]# hosted-engine --set-maintenance --mode=none<br><br><br></div><div>- During monitoring of status, at some point I see &quot;EngineStart&quot; on both host2 and host3<br><br>[root@ovirt02 ~]# hosted-engine --vm-status<br><br><br>--== Host 1 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt01.localdomain.local<br>Host ID                            : 1<br>Engine status                      : {&quot;reason&quot;: &quot;bad vm status&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;down&quot;}<br>Score                              : 3230<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 25cadbfb<br>local_conf_timestamp               : 20055<br>Host timestamp                     : 20040<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=20040 (Sun Jul  9 21:35:55 2017)<br>    host-id=1<br>    score=3230<br>    vm_conf_refresh_time=20055 (Sun Jul  9 21:36:11 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineDown<br>    stopped=False<br><br><br>--== Host 2 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : 192.168.150.103<br>Host ID                            : 2<br>Engine status                      : {&quot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;unknown&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : e6951128<br>local_conf_timestamp               : 3075<br>Host timestamp                     : 3058<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=3058 (Sun Jul  9 21:35:59 2017)<br>    host-id=2<br>    score=3400<br>    vm_conf_refresh_time=3075 (Sun Jul  9 21:36:15 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineStart<br>    stopped=False<br><br><br>--== Host 3 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt03.localdomain.local<br>Host ID                            : 3<br>Engine status                      : {&quot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;unknown&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 382efde5<br>local_conf_timestamp               : 24832<br>Host timestamp                     : 24816<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=24816 (Sun Jul  9 21:36:01 2017)<br>    host-id=3<br>    score=3400<br>    vm_conf_refresh_time=24832 (Sun Jul  9 21:36:17 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineStart<br>    stopped=False<br>[root@ovirt02 ~]# <br><br></div><div>and then<br><br>[root@ovirt02 ~]# hosted-engine --vm-status<br><br><br>--== Host 1 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt01.localdomain.local<br>Host ID                            : 1<br>Engine status                      : {&quot;reason&quot;: &quot;bad vm status&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;down&quot;}<br>Score                              : 3253<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 3fc39f31<br>local_conf_timestamp               : 20087<br>Host timestamp                     : 20070<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=20070 (Sun Jul  9 21:36:26 2017)<br>    host-id=1<br>    score=3253<br>    vm_conf_refresh_time=20087 (Sun Jul  9 21:36:43 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineDown<br>    stopped=False<br><br><br>--== Host 2 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : 192.168.150.103<br>Host ID                            : 2<br>Engine status                      : {&quot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;unknown&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 4a05c31e<br>local_conf_timestamp               : 3109<br>Host timestamp                     : 3079<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=3079 (Sun Jul  9 21:36:19 2017)<br>    host-id=2<br>    score=3400<br>    vm_conf_refresh_time=3109 (Sun Jul  9 21:36:49 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineStarting<br>    stopped=False<br><br><br>--== Host 3 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt03.localdomain.local<br>Host ID                            : 3<br>Engine status                      : {&quot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;unknown&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 382efde5<br>local_conf_timestamp               : 24832<br>Host timestamp                     : 24816<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=24816 (Sun Jul  9 21:36:01 2017)<br>    host-id=3<br>    score=3400<br>    vm_conf_refresh_time=24832 (Sun Jul  9 21:36:17 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineStart<br>    stopped=False<br>[root@ovirt02 ~]# <br><br></div><div>and<br><br>[root@ovirt02 ~]# hosted-engine --vm-status<br><br><br>--== Host 1 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt01.localdomain.local<br>Host ID                            : 1<br>Engine status                      : {&quot;reason&quot;: &quot;bad vm status&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;down&quot;}<br>Score                              : 3253<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 3fc39f31<br>local_conf_timestamp               : 20087<br>Host timestamp                     : 20070<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=20070 (Sun Jul  9 21:36:26 2017)<br>    host-id=1<br>    score=3253<br>    vm_conf_refresh_time=20087 (Sun Jul  9 21:36:43 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineDown<br>    stopped=False<br><br><br>--== Host 2 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : 192.168.150.103<br>Host ID                            : 2<br>Engine status                      : {&quot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;unknown&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 4a05c31e<br>local_conf_timestamp               : 3109<br>Host timestamp                     : 3079<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=3079 (Sun Jul  9 21:36:19 2017)<br>    host-id=2<br>    score=3400<br>    vm_conf_refresh_time=3109 (Sun Jul  9 21:36:49 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineStarting<br>    stopped=False<br><br><br>--== Host 3 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt03.localdomain.local<br>Host ID                            : 3<br>Engine status                      : {&quot;reason&quot;: &quot;vm not running on this host&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;unknown&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : fc1e8cf9<br>local_conf_timestamp               : 24868<br>Host timestamp                     : 24836<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=24836 (Sun Jul  9 21:36:21 2017)<br>    host-id=3<br>    score=3400<br>    vm_conf_refresh_time=24868 (Sun Jul  9 21:36:53 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineStarting<br>    stopped=False<br>[root@ovirt02 ~]# <br><br></div><div>and at the end Host3 goes to &quot;ForceStop&quot; for the engine<br><br>[root@ovirt02 ~]# hosted-engine --vm-status<br><br><br>--== Host 1 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt01.localdomain.local<br>Host ID                            : 1<br>Engine status                      : {&quot;reason&quot;: &quot;bad vm status&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;down&quot;, &quot;detail&quot;: &quot;down&quot;}<br>Score                              : 3312<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : e9d53432<br>local_conf_timestamp               : 20120<br>Host timestamp                     : 20102<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=20102 (Sun Jul  9 21:36:58 2017)<br>    host-id=1<br>    score=3312<br>    vm_conf_refresh_time=20120 (Sun Jul  9 21:37:15 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineDown<br>    stopped=False<br><br><br>--== Host 2 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : 192.168.150.103<br>Host ID                            : 2<br>Engine status                      : {&quot;reason&quot;: &quot;bad vm status&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;up&quot;, &quot;detail&quot;: &quot;powering up&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 7d2330be<br>local_conf_timestamp               : 3141<br>Host timestamp                     : 3124<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=3124 (Sun Jul  9 21:37:04 2017)<br>    host-id=2<br>    score=3400<br>    vm_conf_refresh_time=3141 (Sun Jul  9 21:37:21 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineStarting<br>    stopped=False<br><br><br>--== Host 3 status ==--<br><br>conf_on_shared_storage             : True<br>Status up-to-date                  : True<br>Hostname                           : ovirt03.localdomain.local<br>Host ID                            : 3<br>Engine status                      : {&quot;reason&quot;: &quot;Storage of VM is locked. Is another host already starting the VM?&quot;, &quot;health&quot;: &quot;bad&quot;, &quot;vm&quot;: &quot;already_locked&quot;, &quot;detail&quot;: &quot;down&quot;}<br>Score                              : 3400<br>stopped                            : False<br>Local maintenance                  : False<br>crc32                              : 179825e8<br>local_conf_timestamp               : 24900<br>Host timestamp                     : 24883<br>Extra metadata (valid at timestamp):<br>    metadata_parse_version=1<br>    metadata_feature_version=1<br>    timestamp=24883 (Sun Jul  9 21:37:08 2017)<br>    host-id=3<br>    score=3400<br>    vm_conf_refresh_time=24900 (Sun Jul  9 21:37:24 2017)<br>    conf_on_shared_storage=True<br>    maintenance=False<br>    state=EngineForceStop<br>    stopped=False<br>[root@ovirt02 ~]#<br><br><br></div><div>Comparing /var/log/libvirt/qemu/HostedEngine of host2 and host3<br><br>Host2:<br><br>2017-07-09 19:36:36.094+0000: starting up libvirt version: 2.0.0, package: 10.el7_3.9 (CentOS BuildSystem &lt;<a href="http://bugs.centos.org">http://bugs.centos.org</a>&gt;, 2017-05-25-20:52:28, <a href="http://c1bm.rdu2.centos.org">c1bm.rdu2.centos.org</a>), qemu version: 2.6.0 (qemu-kvm-ev-2.6.0-28.el7.10.1), hostname: ovirt02.localdomain.local<br> ... char device redirected to /dev/pts/1 (label charconsole0)<br>warning: host doesn&#39;t support requested feature: CPUID.07H:EBX.erms [bit 9]<br><br><br></div><div>Host3:<br><br>2017-07-09 19:36:38.143+0000: starting up libvirt version: 2.0.0, package: 10.el7_3.9 (CentOS BuildSystem &lt;<a href="http://bu">http://bu</a><br><a href="http://gs.centos.org">gs.centos.org</a>&gt;, 2017-05-25-20:52:28, <a href="http://c1bm.rdu2.centos.org">c1bm.rdu2.centos.org</a>), qemu version: 2.6.0 (qemu-kvm-ev-2.6.0-28.el7.10.1), hos<br>tname: ovirt03.localdomain.local<br> ... char device redirected to /dev/pts/1 (label charconsole0)<br>2017-07-09 19:36:38.584+0000: shutting down<br>2017-07-09T19:36:38.589729Z qemu-kvm: terminating on signal 15 from pid 1835<br><br></div><div>any comment?<br></div><div>Is it only a matter of powering on the VM in paused mode before starting the OS itself, or do I risk corruption due to 2 qemu-kvm processes trying to start the engine vm os?<br><br></div><div>Thanks,<br></div><div>Gianluca<br></div></div>