Re: [ovirt-users] [hosted-engine] engine VM doesn't respawn when its host was killed (poweroff)

Hi, you have an ISO domain inside the hosted engine VM, don't you? MainThread::INFO::2016-05-04 12:28:47,090::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2016-05-04 12:38:47,504::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/blockSD/d2dad0e9-4f7d-41d6-b61c-487d44ae6d5d/images/157b67ef-1a29-4e51-9396-79d3425b7871/a394b440-91bb-4c7c-b344-146240d66a43 There is a 10 minute gap between two log lines. We log something every 10 seconds.. Please check https://bugzilla.redhat.com/show_bug.cgi?id=1332813 to see if it might be the same issue.

Regards -- Martin Sivak SLA / oVirt On Wed, May 4, 2016 at 8:34 AM, Wee Sritippho <wee.s(a)forest.go.th&gt; wrote: > I've tried again and made sure all hosts have same clock. > > After added all 3 hosts, I tested it by shutting down host01. The engine was > restarted on host02 in less than 2 minutes. I enabled and tested power > management on all hosts (using ilo4), then tried disabling host02's network > to test the fencing. Waited for about 5 minutes and saw in the console that > host02 wasn't fenced. I thought the fencing didn't work and enabled the > network again. host02 was then fenced immediately after the network was > enabled (didn't know why) and the engine was never restarted, even when > host02 is up and running again. I have to start the engine vm manually by > running "hosted-engine --vm-start" on host02. > > I thought it might have something to do with ilo4, so I disabled power > management for all hosts and tried to poweroff host02 again. After about 10 > minutes, the engine still won't start, so I manually start it on host01 > instead. > > Here are my recent actions: > > 2016-05-04 12:25:51 ICT - run hosted-engine --vm-status on host01, vm is > running on host01 > 2016-05-04 12:28:32 ICT - run reboot on host01, engine vm is down > 2016-05-04 12:34:57 ICT - run hosted-engine --vm-status on host01, engine > status on every hosts is "unknown stale-data", host01's score=0, > stopped=true > 2016-05-04 12:37:30 ICT - host01 is pingable > 2016-05-04 12:41:09 ICT - run hosted-engine --vm-status on host02, engine > status on every hosts is "unknown stale-data", all hosts' score=3400, > stopped=false > 2016-05-04 12:43:29 ICT - run hosted-engine --vm-status on host02, vm is > running on host01 > > Log files: https://app.box.com/s/jjgn14onv19e1qi82mkf24jl2baa2l9s > > > On 1/5/2559 19:32, Yedidyah Bar David wrote: >> >> It's very hard to understand your flow when time moves backwards. >> >> Please try again from a clean state. Make sure all hosts have same clock. >> Then document the exact time you do stuff - starting/stopping a host, >> checking status, etc. >> >> Some things to check from your logs: >> >> in agent.host01.log: >> >> MainThread::INFO::2016-04-25 >> >> 15:32:41,370::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) >> Engine down and local host has best score (3400), attempting to start >> engine VM >> ... >> MainThread::INFO::2016-04-25 >> >> 15:32:44,276::hosted_engine::1147::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) >> Engine VM started on localhost >> ... >> MainThread::INFO::2016-04-25 >> >> 15:32:58,478::states::672::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) >> Score is 0 due to unexpected vm shutdown at Mon Apr 25 15:32:58 2016 >> >> Why? >> >> Also, in agent.host03.log: >> >> MainThread::INFO::2016-04-25 >> >> 15:29:53,218::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) >> Engine down and local host has best score (3400), attempting to start >> engine VM >> MainThread::INFO::2016-04-25 >> >> 15:29:53,223::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) >> Trying: notify time=1461572993.22 type=state_transition >> detail=EngineDown-EngineStart hostname='host03.ovirt.forest.go.th' >> MainThread::ERROR::2016-04-25 >> >> 15:30:23,253::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) >> Connection closed: Connection timed out >> >> Why? >> >> Also, in addition to the actions you stated, you changed a lot maintenance >> mode. >> >> You can try something like this to get some interesting lines from >> agent.log: >> >> egrep -i 'start eng|shut|vm started|vm running|vm is running on| >> maintenance detected|migra' >> >> Best, >> >> On Mon, Apr 25, 2016 at 12:27 PM, Wee Sritippho <wee.s(a)forest.go.th&gt; >> wrote: >>> >>> The hosted engine storage is located in an external Fibre Channel SAN. >>> >>> >>> On 25/4/2559 16:19, Martin Sivak wrote: >>>> >>>> Hi, >>>> >>>> it seems that all nodes lost access to storage for some reason after >>>> the host was killed. Where is your hosted engine storage located? >>>> >>>> Regards >>>> >>>> -- >>>> Martin Sivak >>>> SLA / oVirt >>>> >>>> >>>> On Mon, Apr 25, 2016 at 10:58 AM, Wee Sritippho <wee.s(a)forest.go.th&gt; >>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> From the hosted-engine FAQ, the engine VM should be up and running in >>>>> about >>>>> 5 minutes after its host was forced poweroff. However, after updated >>>>> oVirt >>>>> 3.6.4 to 3.6.5, the engine VM won't restart automatically even after >>>>> 10+ >>>>> minutes (I already made sure that global maintenance mode is set to >>>>> none). I >>>>> initially thought its a time sync issue, so I installed and enabled ntp >>>>> on >>>>> the hosts and engine. However, the issue still persists. >>>>> >>>>> ###Versions: >>>>> [root@host01 ~]# rpm -qa | grep ovirt >>>>> libgovirt-0.3.3-1.el7_2.1.x86_64 >>>>> ovirt-vmconsole-1.0.0-1.el7.centos.noarch >>>>> ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch >>>>> ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch >>>>> ovirt-host-deploy-1.4.1-1.el7.centos.noarch >>>>> ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch >>>>> ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch >>>>> ovirt-release36-007-1.noarch >>>>> ovirt-setup-lib-1.0.1-1.el7.centos.noarch >>>>> [root@host01 ~]# rpm -qa | grep vdsm >>>>> vdsm-infra-4.17.26-0.el7.centos.noarch >>>>> vdsm-jsonrpc-4.17.26-0.el7.centos.noarch >>>>> vdsm-gluster-4.17.26-0.el7.centos.noarch >>>>> vdsm-python-4.17.26-0.el7.centos.noarch >>>>> vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch >>>>> vdsm-4.17.26-0.el7.centos.noarch >>>>> vdsm-cli-4.17.26-0.el7.centos.noarch >>>>> vdsm-xmlrpc-4.17.26-0.el7.centos.noarch >>>>> vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch >>>>> >>>>> ###Log files: >>>>> https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r >>>>> >>>>> ###After host02 was killed: >>>>> [root@host03 wees]# hosted-engine --vm-status >>>>> >>>>> >>>>> --== Host 1 status ==-- >>>>> >>>>> Status up-to-date : True >>>>> Hostname : host01.ovirt.forest.go.th >>>>> Host ID : 1 >>>>> Engine status : {"reason": "vm not running on this >>>>> host", "health": "bad", "vm": "down", "detail": "unknown"} >>>>> Score : 3400 >>>>> stopped : False >>>>> Local maintenance : False >>>>> crc32 : 396766e0 >>>>> Host timestamp : 4391 >>>>> >>>>> >>>>> --== Host 2 status ==-- >>>>> >>>>> Status up-to-date : True >>>>> Hostname : host02.ovirt.forest.go.th >>>>> Host ID : 2 >>>>> Engine status : {"health": "good", "vm": "up", >>>>> "detail": "up"} >>>>> Score : 0 >>>>> stopped : True >>>>> Local maintenance : False >>>>> crc32 : 3a345b65 >>>>> Host timestamp : 1458 >>>>> >>>>> >>>>> --== Host 3 status ==-- >>>>> >>>>> Status up-to-date : True >>>>> Hostname : host03.ovirt.forest.go.th >>>>> Host ID : 3 >>>>> Engine status : {"reason": "vm not running on this >>>>> host", "health": "bad", "vm": "down", "detail": "unknown"} >>>>> Score : 3400 >>>>> stopped : False >>>>> Local maintenance : False >>>>> crc32 : 4c34b0ed >>>>> Host timestamp : 11958 >>>>> >>>>> ###After host02 was killed for a while: >>>>> [root@host03 wees]# hosted-engine --vm-status >>>>> >>>>> >>>>> --== Host 1 status ==-- >>>>> >>>>> Status up-to-date : False >>>>> Hostname : host01.ovirt.forest.go.th >>>>> Host ID : 1 >>>>> Engine status : unknown stale-data >>>>> Score : 3400 >>>>> stopped : False >>>>> Local maintenance : False >>>>> crc32 : 72e4e418 >>>>> Host timestamp : 4415 >>>>> >>>>> >>>>> --== Host 2 status ==-- >>>>> >>>>> Status up-to-date : False >>>>> Hostname : host02.ovirt.forest.go.th >>>>> Host ID : 2 >>>>> Engine status : unknown stale-data >>>>> Score : 0 >>>>> stopped : True >>>>> Local maintenance : False >>>>> crc32 : 3a345b65 >>>>> Host timestamp : 1458 >>>>> >>>>> >>>>> --== Host 3 status ==-- >>>>> >>>>> Status up-to-date : False >>>>> Hostname : host03.ovirt.forest.go.th >>>>> Host ID : 3 >>>>> Engine status : unknown stale-data >>>>> Score : 3400 >>>>> stopped : False >>>>> Local maintenance : False >>>>> crc32 : 4c34b0ed >>>>> Host timestamp : 11958 >>>>> >>>>> ###After host02 was up again completely: >>>>> [root@host03 wees]# hosted-engine --vm-status >>>>> >>>>> >>>>> --== Host 1 status ==-- >>>>> >>>>> Status up-to-date : True >>>>> Hostname : host01.ovirt.forest.go.th >>>>> Host ID : 1 >>>>> Engine status : {"reason": "vm not running on this >>>>> host", "health": "bad", "vm": "down", "detail": "unknown"} >>>>> Score : 0 >>>>> stopped : False >>>>> Local maintenance : False >>>>> crc32 : f5728fca >>>>> Host timestamp : 5555 >>>>> >>>>> >>>>> --== Host 2 status ==-- >>>>> >>>>> Status up-to-date : True >>>>> Hostname : host02.ovirt.forest.go.th >>>>> Host ID : 2 >>>>> Engine status : {"health": "good", "vm": "up", >>>>> "detail": "up"} >>>>> Score : 3400 >>>>> stopped : False >>>>> Local maintenance : False >>>>> crc32 : e5284763 >>>>> Host timestamp : 715 >>>>> >>>>> >>>>> --== Host 3 status ==-- >>>>> >>>>> Status up-to-date : True >>>>> Hostname : host03.ovirt.forest.go.th >>>>> Host ID : 3 >>>>> Engine status : {"reason": "vm not running on this >>>>> host", "health": "bad", "vm": "down", "detail": "unknown"} >>>>> Score : 3400 >>>>> stopped : False >>>>> Local maintenance : False >>>>> crc32 : bc10c7fc >>>>> Host timestamp : 13119 >>>>> >>>>> -- >>>>> Wee >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >>> -- >>> วีร์ ศรีทิพโพธิ์ >>> นักวิชาการคอมพิวเตอร์ปฏิบัติการ >>> ศูนย์สารสนเทศ กรมป่าไม้ >>> โทร. 025614292-3 ต่อ 5621 >>> มือถือ. 0864678919 >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> >> > > -- > Wee >