[hosted-engine] engine VM doesn't respawn when its host was killed (poweroff)
Hi, From the hosted-engine FAQ, the engine VM should be up and running in about 5 minutes after its host was forced poweroff. However, after updated oVirt 3.6.4 to 3.6.5, the engine VM won't restart automatically even after 10+ minutes (I already made sure that global maintenance mode is set to none). I initially thought its a time sync issue, so I installed and enabled ntp on the hosts and engine. However, the issue still persists. ###Versions: [root@host01 ~]# rpm -qa | grep ovirt libgovirt-0.3.3-1.el7_2.1.x86_64 ovirt-vmconsole-1.0.0-1.el7.centos.noarch ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch ovirt-host-deploy-1.4.1-1.el7.centos.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch ovirt-release36-007-1.noarch ovirt-setup-lib-1.0.1-1.el7.centos.noarch [root@host01 ~]# rpm -qa | grep vdsm vdsm-infra-4.17.26-0.el7.centos.noarch vdsm-jsonrpc-4.17.26-0.el7.centos.noarch vdsm-gluster-4.17.26-0.el7.centos.noarch vdsm-python-4.17.26-0.el7.centos.noarch vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch vdsm-4.17.26-0.el7.centos.noarch vdsm-cli-4.17.26-0.el7.centos.noarch vdsm-xmlrpc-4.17.26-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch ###Log files: https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r ###After host02 was killed: [root@host03 wees]# hosted-engine --vm-status --== Host 1 status ==-- Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 396766e0 Host timestamp : 4391 --== Host 2 status ==-- Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458 --== Host 3 status ==-- Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958 ###After host02 was killed for a while: [root@host03 wees]# hosted-engine --vm-status --== Host 1 status ==-- Status up-to-date : False Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 72e4e418 Host timestamp : 4415 --== Host 2 status ==-- Status up-to-date : False Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : unknown stale-data Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458 --== Host 3 status ==-- Status up-to-date : False Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958 ###After host02 was up again completely: [root@host03 wees]# hosted-engine --vm-status --== Host 1 status ==-- Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 stopped : False Local maintenance : False crc32 : f5728fca Host timestamp : 5555 --== Host 2 status ==-- Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : e5284763 Host timestamp : 715 --== Host 3 status ==-- Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : bc10c7fc Host timestamp : 13119 -- Wee
Hi, it seems that all nodes lost access to storage for some reason after the host was killed. Where is your hosted engine storage located? Regards -- Martin Sivak SLA / oVirt On Mon, Apr 25, 2016 at 10:58 AM, Wee Sritippho <wee.s@forest.go.th> wrote:
Hi,
From the hosted-engine FAQ, the engine VM should be up and running in about 5 minutes after its host was forced poweroff. However, after updated oVirt 3.6.4 to 3.6.5, the engine VM won't restart automatically even after 10+ minutes (I already made sure that global maintenance mode is set to none). I initially thought its a time sync issue, so I installed and enabled ntp on the hosts and engine. However, the issue still persists.
###Versions: [root@host01 ~]# rpm -qa | grep ovirt libgovirt-0.3.3-1.el7_2.1.x86_64 ovirt-vmconsole-1.0.0-1.el7.centos.noarch ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch ovirt-host-deploy-1.4.1-1.el7.centos.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch ovirt-release36-007-1.noarch ovirt-setup-lib-1.0.1-1.el7.centos.noarch [root@host01 ~]# rpm -qa | grep vdsm vdsm-infra-4.17.26-0.el7.centos.noarch vdsm-jsonrpc-4.17.26-0.el7.centos.noarch vdsm-gluster-4.17.26-0.el7.centos.noarch vdsm-python-4.17.26-0.el7.centos.noarch vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch vdsm-4.17.26-0.el7.centos.noarch vdsm-cli-4.17.26-0.el7.centos.noarch vdsm-xmlrpc-4.17.26-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch
###Log files: https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r
###After host02 was killed: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 396766e0 Host timestamp : 4391
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was killed for a while: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : False Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 72e4e418 Host timestamp : 4415
--== Host 2 status ==--
Status up-to-date : False Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : unknown stale-data Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : False Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was up again completely: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 stopped : False Local maintenance : False crc32 : f5728fca Host timestamp : 5555
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : e5284763 Host timestamp : 715
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : bc10c7fc Host timestamp : 13119
-- Wee
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
The hosted engine storage is located in an external Fibre Channel SAN. On 25/4/2559 16:19, Martin Sivak wrote:
Hi,
it seems that all nodes lost access to storage for some reason after the host was killed. Where is your hosted engine storage located?
Regards
-- Martin Sivak SLA / oVirt
On Mon, Apr 25, 2016 at 10:58 AM, Wee Sritippho <wee.s@forest.go.th> wrote:
Hi,
From the hosted-engine FAQ, the engine VM should be up and running in about 5 minutes after its host was forced poweroff. However, after updated oVirt 3.6.4 to 3.6.5, the engine VM won't restart automatically even after 10+ minutes (I already made sure that global maintenance mode is set to none). I initially thought its a time sync issue, so I installed and enabled ntp on the hosts and engine. However, the issue still persists.
###Versions: [root@host01 ~]# rpm -qa | grep ovirt libgovirt-0.3.3-1.el7_2.1.x86_64 ovirt-vmconsole-1.0.0-1.el7.centos.noarch ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch ovirt-host-deploy-1.4.1-1.el7.centos.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch ovirt-release36-007-1.noarch ovirt-setup-lib-1.0.1-1.el7.centos.noarch [root@host01 ~]# rpm -qa | grep vdsm vdsm-infra-4.17.26-0.el7.centos.noarch vdsm-jsonrpc-4.17.26-0.el7.centos.noarch vdsm-gluster-4.17.26-0.el7.centos.noarch vdsm-python-4.17.26-0.el7.centos.noarch vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch vdsm-4.17.26-0.el7.centos.noarch vdsm-cli-4.17.26-0.el7.centos.noarch vdsm-xmlrpc-4.17.26-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch
###Log files: https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r
###After host02 was killed: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 396766e0 Host timestamp : 4391
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was killed for a while: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : False Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 72e4e418 Host timestamp : 4415
--== Host 2 status ==--
Status up-to-date : False Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : unknown stale-data Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : False Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was up again completely: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 stopped : False Local maintenance : False crc32 : f5728fca Host timestamp : 5555
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : e5284763 Host timestamp : 715
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : bc10c7fc Host timestamp : 13119
-- Wee
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- วีร์ ศรีทิพโพธิ์ นักวิชาการคอมพิวเตอร์ปฏิบัติการ ศูนย์สารสนเทศ กรมป่าไม้ โทร. 025614292-3 ต่อ 5621 มือถือ. 0864678919
It's very hard to understand your flow when time moves backwards. Please try again from a clean state. Make sure all hosts have same clock. Then document the exact time you do stuff - starting/stopping a host, checking status, etc. Some things to check from your logs: in agent.host01.log: MainThread::INFO::2016-04-25 15:32:41,370::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM ... MainThread::INFO::2016-04-25 15:32:44,276::hosted_engine::1147::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost ... MainThread::INFO::2016-04-25 15:32:58,478::states::672::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is 0 due to unexpected vm shutdown at Mon Apr 25 15:32:58 2016 Why? Also, in agent.host03.log: MainThread::INFO::2016-04-25 15:29:53,218::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2016-04-25 15:29:53,223::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1461572993.22 type=state_transition detail=EngineDown-EngineStart hostname='host03.ovirt.forest.go.th' MainThread::ERROR::2016-04-25 15:30:23,253::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Connection closed: Connection timed out Why? Also, in addition to the actions you stated, you changed a lot maintenance mode. You can try something like this to get some interesting lines from agent.log: egrep -i 'start eng|shut|vm started|vm running|vm is running on| maintenance detected|migra' Best, On Mon, Apr 25, 2016 at 12:27 PM, Wee Sritippho <wee.s@forest.go.th> wrote:
The hosted engine storage is located in an external Fibre Channel SAN.
On 25/4/2559 16:19, Martin Sivak wrote:
Hi,
it seems that all nodes lost access to storage for some reason after the host was killed. Where is your hosted engine storage located?
Regards
-- Martin Sivak SLA / oVirt
On Mon, Apr 25, 2016 at 10:58 AM, Wee Sritippho <wee.s@forest.go.th> wrote:
Hi,
From the hosted-engine FAQ, the engine VM should be up and running in about 5 minutes after its host was forced poweroff. However, after updated oVirt 3.6.4 to 3.6.5, the engine VM won't restart automatically even after 10+ minutes (I already made sure that global maintenance mode is set to none). I initially thought its a time sync issue, so I installed and enabled ntp on the hosts and engine. However, the issue still persists.
###Versions: [root@host01 ~]# rpm -qa | grep ovirt libgovirt-0.3.3-1.el7_2.1.x86_64 ovirt-vmconsole-1.0.0-1.el7.centos.noarch ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch ovirt-host-deploy-1.4.1-1.el7.centos.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch ovirt-release36-007-1.noarch ovirt-setup-lib-1.0.1-1.el7.centos.noarch [root@host01 ~]# rpm -qa | grep vdsm vdsm-infra-4.17.26-0.el7.centos.noarch vdsm-jsonrpc-4.17.26-0.el7.centos.noarch vdsm-gluster-4.17.26-0.el7.centos.noarch vdsm-python-4.17.26-0.el7.centos.noarch vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch vdsm-4.17.26-0.el7.centos.noarch vdsm-cli-4.17.26-0.el7.centos.noarch vdsm-xmlrpc-4.17.26-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch
###Log files: https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r
###After host02 was killed: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 396766e0 Host timestamp : 4391
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was killed for a while: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : False Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 72e4e418 Host timestamp : 4415
--== Host 2 status ==--
Status up-to-date : False Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : unknown stale-data Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : False Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was up again completely: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 stopped : False Local maintenance : False crc32 : f5728fca Host timestamp : 5555
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : e5284763 Host timestamp : 715
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : bc10c7fc Host timestamp : 13119
-- Wee
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- วีร์ ศรีทิพโพธิ์ นักวิชาการคอมพิวเตอร์ปฏิบัติการ ศูนย์สารสนเทศ กรมป่าไม้ โทร. 025614292-3 ต่อ 5621 มือถือ. 0864678919
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
I've tried again and made sure all hosts have same clock. After added all 3 hosts, I tested it by shutting down host01. The engine was restarted on host02 in less than 2 minutes. I enabled and tested power management on all hosts (using ilo4), then tried disabling host02's network to test the fencing. Waited for about 5 minutes and saw in the console that host02 wasn't fenced. I thought the fencing didn't work and enabled the network again. host02 was then fenced immediately after the network was enabled (didn't know why) and the engine was never restarted, even when host02 is up and running again. I have to start the engine vm manually by running "hosted-engine --vm-start" on host02. I thought it might have something to do with ilo4, so I disabled power management for all hosts and tried to poweroff host02 again. After about 10 minutes, the engine still won't start, so I manually start it on host01 instead. Here are my recent actions: 2016-05-04 12:25:51 ICT - run hosted-engine --vm-status on host01, vm is running on host01 2016-05-04 12:28:32 ICT - run reboot on host01, engine vm is down 2016-05-04 12:34:57 ICT - run hosted-engine --vm-status on host01, engine status on every hosts is "unknown stale-data", host01's score=0, stopped=true 2016-05-04 12:37:30 ICT - host01 is pingable 2016-05-04 12:41:09 ICT - run hosted-engine --vm-status on host02, engine status on every hosts is "unknown stale-data", all hosts' score=3400, stopped=false 2016-05-04 12:43:29 ICT - run hosted-engine --vm-status on host02, vm is running on host01 Log files: https://app.box.com/s/jjgn14onv19e1qi82mkf24jl2baa2l9s On 1/5/2559 19:32, Yedidyah Bar David wrote:
It's very hard to understand your flow when time moves backwards.
Please try again from a clean state. Make sure all hosts have same clock. Then document the exact time you do stuff - starting/stopping a host, checking status, etc.
Some things to check from your logs:
in agent.host01.log:
MainThread::INFO::2016-04-25 15:32:41,370::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM ... MainThread::INFO::2016-04-25 15:32:44,276::hosted_engine::1147::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost ... MainThread::INFO::2016-04-25 15:32:58,478::states::672::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is 0 due to unexpected vm shutdown at Mon Apr 25 15:32:58 2016
Why?
Also, in agent.host03.log:
MainThread::INFO::2016-04-25 15:29:53,218::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2016-04-25 15:29:53,223::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1461572993.22 type=state_transition detail=EngineDown-EngineStart hostname='host03.ovirt.forest.go.th' MainThread::ERROR::2016-04-25 15:30:23,253::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Connection closed: Connection timed out
Why?
Also, in addition to the actions you stated, you changed a lot maintenance mode.
You can try something like this to get some interesting lines from agent.log:
egrep -i 'start eng|shut|vm started|vm running|vm is running on| maintenance detected|migra'
Best,
On Mon, Apr 25, 2016 at 12:27 PM, Wee Sritippho <wee.s@forest.go.th> wrote:
The hosted engine storage is located in an external Fibre Channel SAN.
On 25/4/2559 16:19, Martin Sivak wrote:
Hi,
it seems that all nodes lost access to storage for some reason after the host was killed. Where is your hosted engine storage located?
Regards
-- Martin Sivak SLA / oVirt
On Mon, Apr 25, 2016 at 10:58 AM, Wee Sritippho <wee.s@forest.go.th> wrote:
Hi,
From the hosted-engine FAQ, the engine VM should be up and running in about 5 minutes after its host was forced poweroff. However, after updated oVirt 3.6.4 to 3.6.5, the engine VM won't restart automatically even after 10+ minutes (I already made sure that global maintenance mode is set to none). I initially thought its a time sync issue, so I installed and enabled ntp on the hosts and engine. However, the issue still persists.
###Versions: [root@host01 ~]# rpm -qa | grep ovirt libgovirt-0.3.3-1.el7_2.1.x86_64 ovirt-vmconsole-1.0.0-1.el7.centos.noarch ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch ovirt-host-deploy-1.4.1-1.el7.centos.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch ovirt-release36-007-1.noarch ovirt-setup-lib-1.0.1-1.el7.centos.noarch [root@host01 ~]# rpm -qa | grep vdsm vdsm-infra-4.17.26-0.el7.centos.noarch vdsm-jsonrpc-4.17.26-0.el7.centos.noarch vdsm-gluster-4.17.26-0.el7.centos.noarch vdsm-python-4.17.26-0.el7.centos.noarch vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch vdsm-4.17.26-0.el7.centos.noarch vdsm-cli-4.17.26-0.el7.centos.noarch vdsm-xmlrpc-4.17.26-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch
###Log files: https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r
###After host02 was killed: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 396766e0 Host timestamp : 4391
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was killed for a while: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : False Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 72e4e418 Host timestamp : 4415
--== Host 2 status ==--
Status up-to-date : False Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : unknown stale-data Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : False Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was up again completely: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 stopped : False Local maintenance : False crc32 : f5728fca Host timestamp : 5555
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : e5284763 Host timestamp : 715
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : bc10c7fc Host timestamp : 13119
-- Wee
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- วีร์ ศรีทิพโพธิ์ นักวิชาการคอมพิวเตอร์ปฏิบัติการ ศูนย์สารสนเทศ กรมป่าไม้ โทร. 025614292-3 ต่อ 5621 มือถือ. 0864678919
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Wee
Hi, you have an ISO domain inside the hosted engine VM, don't you? MainThread::INFO::2016-05-04 12:28:47,090::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2016-05-04 12:38:47,504::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/blockSD/d2dad0e9-4f7d-41d6-b61c-487d44ae6d5d/images/157b67ef-1a29-4e51-9396-79d3425b7871/a394b440-91bb-4c7c-b344-146240d66a43 There is a 10 minute gap between two log lines. We log something every 10 seconds.. Please check https://bugzilla.redhat.com/show_bug.cgi?id=1332813 to see if it might be the same issue. Regards -- Martin Sivak SLA / oVirt On Wed, May 4, 2016 at 8:34 AM, Wee Sritippho <wee.s@forest.go.th> wrote:
I've tried again and made sure all hosts have same clock.
After added all 3 hosts, I tested it by shutting down host01. The engine was restarted on host02 in less than 2 minutes. I enabled and tested power management on all hosts (using ilo4), then tried disabling host02's network to test the fencing. Waited for about 5 minutes and saw in the console that host02 wasn't fenced. I thought the fencing didn't work and enabled the network again. host02 was then fenced immediately after the network was enabled (didn't know why) and the engine was never restarted, even when host02 is up and running again. I have to start the engine vm manually by running "hosted-engine --vm-start" on host02.
I thought it might have something to do with ilo4, so I disabled power management for all hosts and tried to poweroff host02 again. After about 10 minutes, the engine still won't start, so I manually start it on host01 instead.
Here are my recent actions:
2016-05-04 12:25:51 ICT - run hosted-engine --vm-status on host01, vm is running on host01 2016-05-04 12:28:32 ICT - run reboot on host01, engine vm is down 2016-05-04 12:34:57 ICT - run hosted-engine --vm-status on host01, engine status on every hosts is "unknown stale-data", host01's score=0, stopped=true 2016-05-04 12:37:30 ICT - host01 is pingable 2016-05-04 12:41:09 ICT - run hosted-engine --vm-status on host02, engine status on every hosts is "unknown stale-data", all hosts' score=3400, stopped=false 2016-05-04 12:43:29 ICT - run hosted-engine --vm-status on host02, vm is running on host01
Log files: https://app.box.com/s/jjgn14onv19e1qi82mkf24jl2baa2l9s
On 1/5/2559 19:32, Yedidyah Bar David wrote:
It's very hard to understand your flow when time moves backwards.
Please try again from a clean state. Make sure all hosts have same clock. Then document the exact time you do stuff - starting/stopping a host, checking status, etc.
Some things to check from your logs:
in agent.host01.log:
MainThread::INFO::2016-04-25
15:32:41,370::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM ... MainThread::INFO::2016-04-25
15:32:44,276::hosted_engine::1147::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost ... MainThread::INFO::2016-04-25
15:32:58,478::states::672::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is 0 due to unexpected vm shutdown at Mon Apr 25 15:32:58 2016
Why?
Also, in agent.host03.log:
MainThread::INFO::2016-04-25
15:29:53,218::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2016-04-25
15:29:53,223::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1461572993.22 type=state_transition detail=EngineDown-EngineStart hostname='host03.ovirt.forest.go.th' MainThread::ERROR::2016-04-25
15:30:23,253::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Connection closed: Connection timed out
Why?
Also, in addition to the actions you stated, you changed a lot maintenance mode.
You can try something like this to get some interesting lines from agent.log:
egrep -i 'start eng|shut|vm started|vm running|vm is running on| maintenance detected|migra'
Best,
On Mon, Apr 25, 2016 at 12:27 PM, Wee Sritippho <wee.s@forest.go.th> wrote:
The hosted engine storage is located in an external Fibre Channel SAN.
On 25/4/2559 16:19, Martin Sivak wrote:
Hi,
it seems that all nodes lost access to storage for some reason after the host was killed. Where is your hosted engine storage located?
Regards
-- Martin Sivak SLA / oVirt
On Mon, Apr 25, 2016 at 10:58 AM, Wee Sritippho <wee.s@forest.go.th> wrote:
Hi,
From the hosted-engine FAQ, the engine VM should be up and running in about 5 minutes after its host was forced poweroff. However, after updated oVirt 3.6.4 to 3.6.5, the engine VM won't restart automatically even after 10+ minutes (I already made sure that global maintenance mode is set to none). I initially thought its a time sync issue, so I installed and enabled ntp on the hosts and engine. However, the issue still persists.
###Versions: [root@host01 ~]# rpm -qa | grep ovirt libgovirt-0.3.3-1.el7_2.1.x86_64 ovirt-vmconsole-1.0.0-1.el7.centos.noarch ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch ovirt-host-deploy-1.4.1-1.el7.centos.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch ovirt-release36-007-1.noarch ovirt-setup-lib-1.0.1-1.el7.centos.noarch [root@host01 ~]# rpm -qa | grep vdsm vdsm-infra-4.17.26-0.el7.centos.noarch vdsm-jsonrpc-4.17.26-0.el7.centos.noarch vdsm-gluster-4.17.26-0.el7.centos.noarch vdsm-python-4.17.26-0.el7.centos.noarch vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch vdsm-4.17.26-0.el7.centos.noarch vdsm-cli-4.17.26-0.el7.centos.noarch vdsm-xmlrpc-4.17.26-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch
###Log files: https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r
###After host02 was killed: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 396766e0 Host timestamp : 4391
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was killed for a while: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : False Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 72e4e418 Host timestamp : 4415
--== Host 2 status ==--
Status up-to-date : False Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : unknown stale-data Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : False Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was up again completely: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 stopped : False Local maintenance : False crc32 : f5728fca Host timestamp : 5555
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : e5284763 Host timestamp : 715
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : bc10c7fc Host timestamp : 13119
-- Wee
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- วีร์ ศรีทิพโพธิ์ นักวิชาการคอมพิวเตอร์ปฏิบัติการ ศูนย์สารสนเทศ กรมป่าไม้ โทร. 025614292-3 ต่อ 5621 มือถือ. 0864678919
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Wee
On 4 พฤษภาคม 2016 18 นาฬิกา 48 นาที 25 วินาที GMT+07:00, Martin Sivak <msivak@redhat.com> wrote:
Hi,
you have an ISO domain inside the hosted engine VM, don't you?
MainThread::INFO::2016-05-04 12:28:47,090::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2016-05-04 12:38:47,504::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/blockSD/d2dad0e9-4f7d-41d6-b61c-487d44ae6d5d/images/157b67ef-1a29-4e51-9396-79d3425b7871/a394b440-91bb-4c7c-b344-146240d66a43
There is a 10 minute gap between two log lines. We log something every 10 seconds..
Please check https://bugzilla.redhat.com/show_bug.cgi?id=1332813 to see if it might be the same issue.
Yes, exactly the same issue. Thank you.
Regards
-- Martin Sivak SLA / oVirt
I've tried again and made sure all hosts have same clock.
After added all 3 hosts, I tested it by shutting down host01. The engine was restarted on host02 in less than 2 minutes. I enabled and tested
management on all hosts (using ilo4), then tried disabling host02's network to test the fencing. Waited for about 5 minutes and saw in the console that host02 wasn't fenced. I thought the fencing didn't work and enabled
network again. host02 was then fenced immediately after the network was enabled (didn't know why) and the engine was never restarted, even when host02 is up and running again. I have to start the engine vm manually by running "hosted-engine --vm-start" on host02.
I thought it might have something to do with ilo4, so I disabled
management for all hosts and tried to poweroff host02 again. After about 10 minutes, the engine still won't start, so I manually start it on host01 instead.
Here are my recent actions:
2016-05-04 12:25:51 ICT - run hosted-engine --vm-status on host01, vm is running on host01 2016-05-04 12:28:32 ICT - run reboot on host01, engine vm is down 2016-05-04 12:34:57 ICT - run hosted-engine --vm-status on host01, engine status on every hosts is "unknown stale-data", host01's score=0, stopped=true 2016-05-04 12:37:30 ICT - host01 is pingable 2016-05-04 12:41:09 ICT - run hosted-engine --vm-status on host02, engine status on every hosts is "unknown stale-data", all hosts' score=3400, stopped=false 2016-05-04 12:43:29 ICT - run hosted-engine --vm-status on host02, vm is running on host01
Log files: https://app.box.com/s/jjgn14onv19e1qi82mkf24jl2baa2l9s
On 1/5/2559 19:32, Yedidyah Bar David wrote:
It's very hard to understand your flow when time moves backwards.
Please try again from a clean state. Make sure all hosts have same
clock.
Then document the exact time you do stuff - starting/stopping a host, checking status, etc.
Some things to check from your logs:
in agent.host01.log:
MainThread::INFO::2016-04-25
15:32:41,370::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine down and local host has best score (3400), attempting to start engine VM ... MainThread::INFO::2016-04-25
15:32:44,276::hosted_engine::1147::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm)
Engine VM started on localhost ... MainThread::INFO::2016-04-25
15:32:58,478::states::672::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score)
Score is 0 due to unexpected vm shutdown at Mon Apr 25 15:32:58 2016
Why?
Also, in agent.host03.log:
MainThread::INFO::2016-04-25
15:29:53,218::states::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2016-04-25
15:29:53,223::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Trying: notify time=1461572993.22 type=state_transition detail=EngineDown-EngineStart hostname='host03.ovirt.forest.go.th' MainThread::ERROR::2016-04-25
15:30:23,253::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate)
Connection closed: Connection timed out
Why?
Also, in addition to the actions you stated, you changed a lot
On Wed, May 4, 2016 at 8:34 AM, Wee Sritippho <wee.s@forest.go.th> wrote: power the power maintenance
mode.
You can try something like this to get some interesting lines from agent.log:
egrep -i 'start eng|shut|vm started|vm running|vm is running on| maintenance detected|migra'
Best,
On Mon, Apr 25, 2016 at 12:27 PM, Wee Sritippho <wee.s@forest.go.th> wrote:
The hosted engine storage is located in an external Fibre Channel
SAN.
On 25/4/2559 16:19, Martin Sivak wrote:
Hi,
it seems that all nodes lost access to storage for some reason
after
the host was killed. Where is your hosted engine storage located?
Regards
-- Martin Sivak SLA / oVirt
On Mon, Apr 25, 2016 at 10:58 AM, Wee Sritippho <wee.s@forest.go.th> wrote:
Hi,
From the hosted-engine FAQ, the engine VM should be up and
running in
about 5 minutes after its host was forced poweroff. However, after updated oVirt 3.6.4 to 3.6.5, the engine VM won't restart automatically even after 10+ minutes (I already made sure that global maintenance mode is set to none). I initially thought its a time sync issue, so I installed and enabled ntp on the hosts and engine. However, the issue still persists.
###Versions: [root@host01 ~]# rpm -qa | grep ovirt libgovirt-0.3.3-1.el7_2.1.x86_64 ovirt-vmconsole-1.0.0-1.el7.centos.noarch ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch ovirt-hosted-engine-ha-1.3.5.3-1.el7.centos.noarch ovirt-host-deploy-1.4.1-1.el7.centos.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7.centos.noarch ovirt-hosted-engine-setup-1.3.5.0-1.el7.centos.noarch ovirt-release36-007-1.noarch ovirt-setup-lib-1.0.1-1.el7.centos.noarch [root@host01 ~]# rpm -qa | grep vdsm vdsm-infra-4.17.26-0.el7.centos.noarch vdsm-jsonrpc-4.17.26-0.el7.centos.noarch vdsm-gluster-4.17.26-0.el7.centos.noarch vdsm-python-4.17.26-0.el7.centos.noarch vdsm-yajsonrpc-4.17.26-0.el7.centos.noarch vdsm-4.17.26-0.el7.centos.noarch vdsm-cli-4.17.26-0.el7.centos.noarch vdsm-xmlrpc-4.17.26-0.el7.centos.noarch vdsm-hook-vmfex-dev-4.17.26-0.el7.centos.noarch
###Log files: https://app.box.com/s/fkurmwagogwkv5smkwwq7i4ztmwf9q9r
###After host02 was killed: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 396766e0 Host timestamp : 4391
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was killed for a while: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : False Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 72e4e418 Host timestamp : 4415
--== Host 2 status ==--
Status up-to-date : False Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : unknown stale-data Score : 0 stopped : True Local maintenance : False crc32 : 3a345b65 Host timestamp : 1458
--== Host 3 status ==--
Status up-to-date : False Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 4c34b0ed Host timestamp : 11958
###After host02 was up again completely: [root@host03 wees]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True Hostname : host01.ovirt.forest.go.th Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 stopped : False Local maintenance : False crc32 : f5728fca Host timestamp : 5555
--== Host 2 status ==--
Status up-to-date : True Hostname : host02.ovirt.forest.go.th Host ID : 2 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 3400 stopped : False Local maintenance : False crc32 : e5284763 Host timestamp : 715
--== Host 3 status ==--
Status up-to-date : True Hostname : host03.ovirt.forest.go.th Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : bc10c7fc Host timestamp : 13119
-- Wee
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- วีร์ ศรีทิพโพธิ์ นักวิชาการคอมพิวเตอร์ปฏิบัติการ ศูนย์สารสนเทศ กรมป่าไม้ โทร. 025614292-3 ต่อ 5621 มือถือ. 0864678919
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Wee
-- วีร์ ศรีทิพโพธิ์ นักวิชาการคอมพิวเตอร์ปฏิบัติการ ศูนย์สารสนเทศ กรมป่าไม้ โทร. 025614292-3 ต่อ 5621 มือถือ. 0864678919
participants (3)
-
Martin Sivak -
Wee Sritippho -
Yedidyah Bar David