[ovirt-users] [hosted-engine] engine VM doesn't respawn when its host was killed (poweroff)
Wee Sritippho
wee.s at forest.go.th
Wed May 4 06:34:12 UTC 2016
I've tried again and made sure all hosts have same clock.
After added all 3 hosts, I tested it by shutting down host01. The engine
was restarted on host02 in less than 2 minutes. I enabled and tested
power management on all hosts (using ilo4), then tried disabling
host02's network to test the fencing. Waited for about 5 minutes and saw
in the console that host02 wasn't fenced. I thought the fencing didn't
work and enabled the network again. host02 was then fenced immediately
after the network was enabled (didn't know why) and the engine was never
restarted, even when host02 is up and running again. I have to start the
engine vm manually by running "hosted-engine --vm-start" on host02.
I thought it might have something to do with ilo4, so I disabled power
management for all hosts and tried to poweroff host02 again. After about
10 minutes, the engine still won't start, so I manually start it on
host01 instead.
Here are my recent actions:
2016-05-04 12:25:51 ICT - run hosted-engine --vm-status on host01, vm is
running on host01
2016-05-04 12:28:32 ICT - run reboot on host01, engine vm is down
2016-05-04 12:34:57 ICT - run hosted-engine --vm-status on host01,
engine status on every hosts is "unknown stale-data", host01's score=0,
stopped=true
2016-05-04 12:37:30 ICT - host01 is pingable
2016-05-04 12:41:09 ICT - run hosted-engine --vm-status on host02,
engine status on every hosts is "unknown stale-data", all hosts'
score=3400, stopped=false
2016-05-04 12:43:29 ICT - run hosted-engine --vm-status on host02, vm is
running on host01
Log files: https://app.box.com/s/jjgn14onv19e1qi82mkf24jl2baa2l9s
On 1/5/2559 19:32, Yedidyah Bar David wrote:
> It's very hard to understand your flow when time moves backwards.
>
> Please try again from a clean state. Make sure all hosts have same clock.
> Then document the exact time you do stuff - starting/stopping a host,
> checking status, etc.
>
> Some things to check from your logs:
>
> in agent.host01.log:
>
> MainThread::INFO::2016-04-25
> 15:32:41,370::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine down and local host has best score (3400), attempting to start
> engine VM
> ...
> MainThread::INFO::2016-04-25
> 15:32:44,276::hosted_engine::1147::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)
> Engine VM started on localhost
> ...
> MainThread::INFO::2016-04-25
> 15:32:58,478::states::672::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score)
> Score is 0 due to unexpected vm shutdown at Mon Apr 25 15:32:58 2016
>
> Why?
>
> Also, in agent.host03.log:
>
> MainThread::INFO::2016-04-25
> 15:29:53,218::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
> Engine down and local host has best score (3400), attempting to start
> engine VM
> MainThread::INFO::2016-04-25
> 15:29:53,223::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
> Trying: notify time=1461572993.22 type=state_transition
> detail=EngineDown-EngineStart hostname='host03.ovirt.forest.go.th'
> MainThread::ERROR::2016-04-25
> 15:30:23,253::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate)
> Connection closed: Connection timed out
>
> Why?
>
> Also, in addition to the actions you stated, you changed a lot maintenance mode.
>
> You can try something like this to get some interesting lines from agent.log:
>
> egrep -i 'start eng|shut|vm started|vm running|vm is running on|
> maintenance detected|migra'
>
> Best,
>
> On Mon, Apr 25, 2016 at 12:27 PM, Wee Sritippho <wee.s at forest.go.th> wrote:
>> The hosted engine storage is located in an external Fibre Channel SAN.
>>
>>
>> On 25/4/2559 16:19, Martin Sivak wrote:
>>> Hi,
>>>
>>> it seems that all nodes lost access to storage for some reason after
>>> the host was killed. Where is your hosted engine storage located?
>>>
>>> Regards
>>>
>>> --
>>> Martin Sivak
>>> SLA / oVirt
>>>
>>>
>>> On Mon, Apr 25, 2016 at 10:58 AM, Wee Sritippho <wee.s at forest.go.th>
>>> wrote:
>>>> Hi,
>>>>
>>>> From the hosted-engine FAQ, the engine VM should be up and running in
>>>> about
>>>> 5 minutes after its host was forced poweroff. However, after updated
>>>> oVirt
>>>> 3.6.4 to 3.6.5, the engine VM won't restart automatically even after 10+
>>>> minutes (I already made sure that global maintenance mode is set to
>>>> none). I
>>>> initially thought its a time sync issue, so I installed and enabled ntp
>>>> on
>>>> the hosts and engine. However, the issue still persists.
>>>>
>>>> ###Versions:
>>>> [root at host01 ~]# rpm -qa | grep ovirt
>>>> libgovirt-0.3.3-1.el7_2.1.x86_64
>>>> ovirt-vmconsole-1.0.0-1.el7.centos.noarch
>>>> ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch
>>>> ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch
>>>> ovirt-host-deploy-1.4.1-1.el7.centos.noarch
>>>> ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch
>>>> ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch
>>>> ovirt-release36-007-1.noarch
>>>> ovirt-setup-lib-1.0.1-1.el7.centos.noarch
>>>> [root at host01 ~]# rpm -qa | grep vdsm
>>>> vdsm-infra-4.17.26-0.el7.centos.noarch
>>>> vdsm-jsonrpc-4.17.26-0.el7.centos.noarch
>>>> vdsm-gluster-4.17.26-0.el7.centos.noarch
>>>> vdsm-python-4.17.26-0.el7.centos.noarch
>>>> vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch
>>>> vdsm-4.17.26-0.el7.centos.noarch
>>>> vdsm-cli-4.17.26-0.el7.centos.noarch
>>>> vdsm-xmlrpc-4.17.26-0.el7.centos.noarch
>>>> vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch
>>>>
>>>> ###Log files:
>>>> https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r
>>>>
>>>> ###After host02 was killed:
>>>> [root at host03 wees]# hosted-engine --vm-status
>>>>
>>>>
>>>> --== Host 1 status ==--
>>>>
>>>> Status up-to-date : True
>>>> Hostname : host01.ovirt.forest.go.th
>>>> Host ID : 1
>>>> Engine status : {"reason": "vm not running on this
>>>> host", "health": "bad", "vm": "down", "detail": "unknown"}
>>>> Score : 3400
>>>> stopped : False
>>>> Local maintenance : False
>>>> crc32 : 396766e0
>>>> Host timestamp : 4391
>>>>
>>>>
>>>> --== Host 2 status ==--
>>>>
>>>> Status up-to-date : True
>>>> Hostname : host02.ovirt.forest.go.th
>>>> Host ID : 2
>>>> Engine status : {"health": "good", "vm": "up",
>>>> "detail": "up"}
>>>> Score : 0
>>>> stopped : True
>>>> Local maintenance : False
>>>> crc32 : 3a345b65
>>>> Host timestamp : 1458
>>>>
>>>>
>>>> --== Host 3 status ==--
>>>>
>>>> Status up-to-date : True
>>>> Hostname : host03.ovirt.forest.go.th
>>>> Host ID : 3
>>>> Engine status : {"reason": "vm not running on this
>>>> host", "health": "bad", "vm": "down", "detail": "unknown"}
>>>> Score : 3400
>>>> stopped : False
>>>> Local maintenance : False
>>>> crc32 : 4c34b0ed
>>>> Host timestamp : 11958
>>>>
>>>> ###After host02 was killed for a while:
>>>> [root at host03 wees]# hosted-engine --vm-status
>>>>
>>>>
>>>> --== Host 1 status ==--
>>>>
>>>> Status up-to-date : False
>>>> Hostname : host01.ovirt.forest.go.th
>>>> Host ID : 1
>>>> Engine status : unknown stale-data
>>>> Score : 3400
>>>> stopped : False
>>>> Local maintenance : False
>>>> crc32 : 72e4e418
>>>> Host timestamp : 4415
>>>>
>>>>
>>>> --== Host 2 status ==--
>>>>
>>>> Status up-to-date : False
>>>> Hostname : host02.ovirt.forest.go.th
>>>> Host ID : 2
>>>> Engine status : unknown stale-data
>>>> Score : 0
>>>> stopped : True
>>>> Local maintenance : False
>>>> crc32 : 3a345b65
>>>> Host timestamp : 1458
>>>>
>>>>
>>>> --== Host 3 status ==--
>>>>
>>>> Status up-to-date : False
>>>> Hostname : host03.ovirt.forest.go.th
>>>> Host ID : 3
>>>> Engine status : unknown stale-data
>>>> Score : 3400
>>>> stopped : False
>>>> Local maintenance : False
>>>> crc32 : 4c34b0ed
>>>> Host timestamp : 11958
>>>>
>>>> ###After host02 was up again completely:
>>>> [root at host03 wees]# hosted-engine --vm-status
>>>>
>>>>
>>>> --== Host 1 status ==--
>>>>
>>>> Status up-to-date : True
>>>> Hostname : host01.ovirt.forest.go.th
>>>> Host ID : 1
>>>> Engine status : {"reason": "vm not running on this
>>>> host", "health": "bad", "vm": "down", "detail": "unknown"}
>>>> Score : 0
>>>> stopped : False
>>>> Local maintenance : False
>>>> crc32 : f5728fca
>>>> Host timestamp : 5555
>>>>
>>>>
>>>> --== Host 2 status ==--
>>>>
>>>> Status up-to-date : True
>>>> Hostname : host02.ovirt.forest.go.th
>>>> Host ID : 2
>>>> Engine status : {"health": "good", "vm": "up",
>>>> "detail": "up"}
>>>> Score : 3400
>>>> stopped : False
>>>> Local maintenance : False
>>>> crc32 : e5284763
>>>> Host timestamp : 715
>>>>
>>>>
>>>> --== Host 3 status ==--
>>>>
>>>> Status up-to-date : True
>>>> Hostname : host03.ovirt.forest.go.th
>>>> Host ID : 3
>>>> Engine status : {"reason": "vm not running on this
>>>> host", "health": "bad", "vm": "down", "detail": "unknown"}
>>>> Score : 3400
>>>> stopped : False
>>>> Local maintenance : False
>>>> crc32 : bc10c7fc
>>>> Host timestamp : 13119
>>>>
>>>> --
>>>> Wee
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>
>> --
>> วีร์ ศรีทิพโพธิ์
>> นักวิชาการคอมพิวเตอร์ปฏิบัติการ
>> ศูนย์สารสนเทศ กรมป่าไม้
>> โทร. 025614292-3 ต่อ 5621
>> มือถือ. 0864678919
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
>
--
Wee
More information about the Users
mailing list