12:54 p.m.
------=_Part_1882534_1428653136.1419879250032
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi,
Can you please provide engine.log from /var/log/ovirt-engine/engine.log and to try as
follows:
1. Revert all tree hosts to maintenance-mode=none.
2. Check that engine up and running.
3. Turn one of the hosts that is not running the engine, to maintenance mode local.
4. Turn host that is running the engine to maintenance mode local.
5. Check that engine migrated to one and the only remaining host, that had not been
put in to maintenance mode at all.
Can you also provide your engine version, is it 3.4 something?
Thanks in advance.
Best regards,
Nikolai
____________________
Nikolai Sednev
Senior Quality Engineer at Compute team
Red Hat Israel
34 Jerusalem Road,
Ra'anana, Israel 43501
Tel: +972 9 7692043
Mobile: +972 52 7342734
Email: nsednev(a)redhat.com
IRC: nsednev
----- Original Message -----
From: users-request(a)ovirt.org
To: users(a)ovirt.org
Sent: Monday, December 29, 2014 8:29:36 PM
Subject: Users Digest, Vol 39, Issue 171
Send Users mailing list submissions to
users(a)ovirt.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.ovirt.org/mailman/listinfo/users
or, via email, send a message with subject or body 'help' to
users-request(a)ovirt.org
You can reach the person managing the list at
users-owner(a)ovirt.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Users digest..."
Today's Topics:
1. Re: VM failover with ovirt3.5 (Yue, Cong)
----------------------------------------------------------------------
Message: 1
Date: Mon, 29 Dec 2014 10:29:04 -0800
From: "Yue, Cong" <Cong_Yue(a)alliedtelesis.com>
To: Artyom Lukianov <alukiano(a)redhat.com>
Cc: "users(a)ovirt.org" <users(a)ovirt.org>
Subject: Re: [ovirt-users] VM failover with ovirt3.5
Message-ID: <21D302CF-AD6F-4E8C-A373-52ADAC1C129B(a)alliedtelesis.com>
Content-Type: text/plain; charset="utf-8"
I disabled local maintenance mode for all hosts, and then only set the host where HE VM is
there to local maintenance mode. The logs are as follows. During the migration of HE VM ,
it shows some fatal error happen. By the way, also HE VM can not work with live migration.
Instead, other VMs can do live migration.
---
[root@compute2-3 ~]# hosted-engine --set-maintenance --mode=local
You have new mail in /var/spool/mail/root
[root@compute2-3 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log
MainThread::INFO::2014-12-29
13:16:12,435::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.92 (id: 3, score: 2400)
MainThread::INFO::2014-12-29
13:16:22,711::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-29
13:16:22,711::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.92 (id: 3, score: 2400)
MainThread::INFO::2014-12-29
13:16:32,978::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-29
13:16:32,978::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-29
13:16:43,272::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-29
13:16:43,272::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-29
13:16:53,316::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine vm running on localhost
MainThread::INFO::2014-12-29
13:16:53,562::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-29
13:16:53,562::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-29
13:17:03,600::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-29
13:17:03,611::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Trying: notify time=1419877023.61 type=state_transition
detail=EngineUp-LocalMaintenanceMigrateVm hostname='compute2-3'
MainThread::INFO::2014-12-29
13:17:03,672::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Success, was notification of state_transition
(EngineUp-LocalMaintenanceMigrateVm) sent? sent
MainThread::INFO::2014-12-29
13:17:03,911::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score)
Score is 0 due to local maintenance mode
MainThread::INFO::2014-12-29
13:17:03,912::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenanceMigrateVm (score: 0)
MainThread::INFO::2014-12-29
13:17:03,912::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-29
13:17:03,960::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Trying: notify time=1419877023.96 type=state_transition
detail=LocalMaintenanceMigrateVm-EngineMigratingAway
hostname='compute2-3'
MainThread::INFO::2014-12-29
13:17:03,980::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Success, was notification of state_transition
(LocalMaintenanceMigrateVm-EngineMigratingAway) sent? sent
MainThread::INFO::2014-12-29
13:17:04,218::states::66::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_penalize_memory)
Penalizing score by 400 due to low free memory
MainThread::INFO::2014-12-29
13:17:04,218::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineMigratingAway (score: 2000)
MainThread::INFO::2014-12-29
13:17:04,219::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::ERROR::2014-12-29
13:17:14,251::hosted_engine::867::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitor_migration)
Failed to migrate
Traceback (most recent call last):
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 863, in _monitor_migration
vm_id,
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/vds_client.py",
line 85, in run_vds_client_cmd
response['status']['message'])
DetailedError: Error 12 from migrateStatus: Fatal error during migration
MainThread::INFO::2014-12-29
13:17:14,262::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Trying: notify time=1419877034.26 type=state_transition
detail=EngineMigratingAway-ReinitializeFSM hostname='compute2-3'
MainThread::INFO::2014-12-29
13:17:14,263::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Success, was notification of state_transition
(EngineMigratingAway-ReinitializeFSM) sent? ignored
MainThread::INFO::2014-12-29
13:17:14,496::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state ReinitializeFSM (score: 0)
MainThread::INFO::2014-12-29
13:17:14,496::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-29
13:17:24,536::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-29
13:17:24,547::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Trying: notify time=1419877044.55 type=state_transition
detail=ReinitializeFSM-LocalMaintenance hostname='compute2-3'
MainThread::INFO::2014-12-29
13:17:24,574::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Success, was notification of state_transition
(ReinitializeFSM-LocalMaintenance) sent? sent
MainThread::INFO::2014-12-29
13:17:24,812::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-29
13:17:24,812::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-29
13:17:34,851::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-29
13:17:35,095::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-29
13:17:35,095::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-29
13:17:45,130::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-29
13:17:45,368::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-29
13:17:45,368::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
^C
[root@compute2-3 ~]#
[root@compute2-3 ~]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True
Hostname : 10.0.0.94
Host ID : 1
Engine status : {"health": "good", "vm": "up",
"detail": "up"}
Score : 0
Local maintenance : True
Host timestamp : 1014956<tel:1014956>
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1014956<tel:1014956> (Mon Dec 29 13:20:19 2014)
host-id=1
score=0
maintenance=True
state=LocalMaintenance
--== Host 2 status ==--
Status up-to-date : True
Hostname : 10.0.0.93
Host ID : 2
Engine status : {"reason": "vm not running on
this host", "health": "bad", "vm": "down",
"detail": "unknown"}
Score : 2400
Local maintenance : False
Host timestamp : 866019
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=866019 (Mon Dec 29 10:19:45 2014)
host-id=2
score=2400
maintenance=False
state=EngineDown
--== Host 3 status ==--
Status up-to-date : True
Hostname : 10.0.0.92
Host ID : 3
Engine status : {"reason": "vm not running on
this host", "health": "bad", "vm": "down",
"detail": "unknown"}
Score : 2400
Local maintenance : False
Host timestamp : 860493
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=860493 (Mon Dec 29 10:20:35 2014)
host-id=3
score=2400
maintenance=False
state=EngineDown
[root@compute2-3 ~]#
---
Thanks,
Cong
On 2014/12/29, at 8:43, "Artyom Lukianov"
<alukiano@redhat.com<mailto:alukiano@redhat.com>> wrote:
I see that HE vm run on host with ip 10.0.0.94, and two another hosts in "Local
Maintenance" state, so vm will not migrate to any of them, can you try disable local
maintenance on all hosts in HE environment and after enable "local maintenance"
on host where HE vm run, and provide also output of hosted-engine --vm-status.
Failover works in next way:
1) if host where run HE vm have score less by 800 that some other host in HE environment,
HE vm will migrate on host with best score
2) if something happen to vm(kernel panic, crash of service...), agent will restart HE vm
on another host in HE environment with positive score
3) if put to local maintenance host with HE vm, vm will migrate to another host with
positive score
Thanks.
----- Original Message -----
From: "Cong Yue"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com>>
To: "Artyom Lukianov"
<alukiano@redhat.com<mailto:alukiano@redhat.com>>
Cc: "Simone Tiraboschi"
<stirabos@redhat.com<mailto:stirabos@redhat.com>>,
users@ovirt.org<mailto:users@ovirt.org>
Sent: Monday, December 29, 2014 6:30:42 PM
Subject: Re: [ovirt-users] VM failover with ovirt3.5
Thanks and the --vm-status log is as follows:
[root@compute2-2 ~]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True
Hostname : 10.0.0.94
Host ID : 1
Engine status : {"health": "good", "vm": "up",
"detail": "up"}
Score : 2400
Local maintenance : False
Host timestamp : 1008087
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1008087<tel:1008087> (Mon Dec 29 11:25:51 2014)
host-id=1
score=2400
maintenance=False
state=EngineUp
--== Host 2 status ==--
Status up-to-date : True
Hostname : 10.0.0.93
Host ID : 2
Engine status : {"reason": "vm not running on
this host", "health": "bad", "vm": "down",
"detail": "unknown"}
Score : 0
Local maintenance : True
Host timestamp : 859142
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=859142 (Mon Dec 29 08:25:08 2014)
host-id=2
score=0
maintenance=True
state=LocalMaintenance
--== Host 3 status ==--
Status up-to-date : True
Hostname : 10.0.0.92
Host ID : 3
Engine status : {"reason": "vm not running on
this host", "health": "bad", "vm": "down",
"detail": "unknown"}
Score : 0
Local maintenance : True
Host timestamp : 853615
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=853615 (Mon Dec 29 08:25:57 2014)
host-id=3
score=0
maintenance=True
state=LocalMaintenance
You have new mail in /var/spool/mail/root
[root@compute2-2 ~]#
Could you please explain how VM failover works inside ovirt? Is there any other debug
option I can enable to check the problem?
Thanks,
Cong
On 2014/12/29, at 1:39, "Artyom Lukianov"
<alukiano@redhat.com<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>>
wrote:
Can you also provide output of hosted-engine --vm-status please, previous time it was
useful, because I do not see something unusual.
Thanks
----- Original Message -----
From: "Cong Yue"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>>
To: "Artyom Lukianov"
<alukiano@redhat.com<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>>
Cc: "Simone Tiraboschi"
<stirabos@redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>>,
users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org>
Sent: Monday, December 29, 2014 7:15:24 AM
Subject: Re: [ovirt-users] VM failover with ovirt3.5
Also I change the maintenance mode to local in another host. But also the VM in this host
can not be migrated. The logs are as follows.
[root@compute2-2 ~]# hosted-engine --set-maintenance --mode=local
[root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log
MainThread::INFO::2014-12-28
21:09:04,184::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-28
21:09:14,603::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-28
21:09:14,603::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-28
21:09:24,903::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-28
21:09:24,904::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-28
21:09:35,026::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine vm is running on host 10.0.0.94 (id 1)
MainThread::INFO::2014-12-28
21:09:35,236::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-28
21:09:35,236::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-28
21:09:45,604::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-28
21:09:45,604::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-28
21:09:55,691::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-28
21:09:55,701::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Trying: notify time=1419829795.7 type=state_transition
detail=EngineDown-LocalMaintenance hostname='compute2-2'
MainThread::INFO::2014-12-28
21:09:55,761::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
Success, was notification of state_transition
(EngineDown-LocalMaintenance) sent? sent
MainThread::INFO::2014-12-28
21:09:55,990::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score)
Score is 0 due to local maintenance mode
MainThread::INFO::2014-12-28
21:09:55,990::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-28
21:09:55,991::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
^C
You have new mail in /var/spool/mail/root
[root@compute2-2 ~]# ps -ef | grep qemu
root 18420 2777 0 21:10<x-apple-data-detectors://39> pts/0
00:00:00<x-apple-data-detectors://40> grep --color=auto qemu
qemu 29809 1 0 Dec19 ? 01:17:20 /usr/libexec/qemu-kvm
-name testvm2-2 -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Nehalem
-m 500 -realtime mlock=off -smp
1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid
c31e97d0-135e-42da-9954-162b5228dce3 -smbios
type=1,manufacturer=oVirt,product=oVirt
Node,version=7-0.1406.el7.centos.2.5,serial=4C4C4544-0059-3610-8033-B4C04F395931,uuid=c31e97d0-135e-42da-9954-162b5228dce3
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/testvm2-2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=2014-12-19T20:17:17<x-apple-data-detectors://42>,driftfix=slew
-no-kvm-pit-reinjection
-no-hpet -no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-drive
file=/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/5cbeb8c9-4f04-48d0-a5eb-78c49187c550/a0570e8c-9867-4ec4-818f-11e102fc4f9b,if=none,id=drive-virtio-disk0,format=qcow2,serial=5cbeb8c9-4f04-48d0-a5eb-78c49187c550,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:db:94:00,bus=pci.0,addr=0x3
-chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3.com.redhat.rhevm.vdsm,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
-chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3.org.qemu.guest_agent.0,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-chardev spicevmc,id=charchannel2,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-spice
tls-port=5901,addr=10.0.0.93,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on
-k en-us -vga qxl -global qxl-vga.ram_size=67108864<tel:67108864> -global
qxl-vga.vram_size=33554432<tel:33554432> -incoming tcp:[::]:49152 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
[root@compute2-2 ~]#
Thanks,
Cong
On 2014/12/28, at 20:53, "Yue, Cong"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>>
wrote:
I checked it again and confirmed there is one guest VM is running on the top of this host.
The log is as follows:
[root@compute2-1 vdsm]# ps -ef | grep qemu
qemu 2983 846 0 Dec19 ? 00:00:00<x-apple-data-detectors://0> [supervdsmServer]
<defunct>
root 5489 3053 0 20:49<x-apple-data-detectors://1> pts/0
00:00:00<x-apple-data-detectors://2> grep --color=auto qemu
qemu 26128 1 0 Dec19 ? 01:09:19 /usr/libexec/qemu-kvm
-name testvm2 -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Nehalem -m
500 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1
-uuid e46bca87-4df5-4287-844b-90a26fccef33 -smbios
type=1,manufacturer=oVirt,product=oVirt
Node,version=7-0.1406.el7.centos.2.5,serial=4C4C4544-0030-3310-8059-B8C04F585231,uuid=e46bca87-4df5-4287-844b-90a26fccef33
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/testvm2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=2014-12-19T20:18:01<x-apple-data-detectors://4>,driftfix=slew
-no-kvm-pit-reinjection
-no-hpet -no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial=
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-drive
file=/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/b4b5426b-95e3-41af-b286-da245891cdaf/0f688d49-97e3-4f1d-84d4-ac1432d903b3,if=none,id=drive-virtio-disk0,format=qcow2,serial=b4b5426b-95e3-41af-b286-da245891cdaf,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:db:94:01,bus=pci.0,addr=0x3
-chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef33.com.redhat.rhevm.vdsm,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
-chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef33.org.qemu.guest_agent.0,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-chardev spicevmc,id=charchannel2,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-spice
tls-port=5900,addr=10.0.0.92,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on
-k en-us -vga qxl -global qxl-vga.ram_size=67108864<tel:67108864> -global
qxl-vga.vram_size=33554432<tel:33554432> -incoming tcp:[::]:49152 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7
[root@compute2-1 vdsm]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log
MainThread::INFO::2014-12-28
20:49:27,315::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-28
20:49:27,646::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-28
20:49:27,646::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-28
20:49:37,732::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-28
20:49:37,961::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-28
20:49:37,961::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-28
20:49:48,048::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-28
20:49:48,319::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score)
Score is 0 due to local maintenance mode
MainThread::INFO::2014-12-28
20:49:48,319::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-28
20:49:48,319::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
Thanks,
Cong
On 2014/12/28, at 3:46, "Artyom Lukianov"
<alukiano@redhat.com<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>>
wrote:
I see that you set local maintenance on host3 that do not have engine vm on it, so it
nothing to migrate from this host.
If you set local maintenance on host1, vm must migrate to another host with positive
score.
Thanks
----- Original Message -----
From: "Cong Yue"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>>
To: "Simone Tiraboschi"
<stirabos@redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>>
Cc:
users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org>
Sent: Saturday, December 27, 2014 6:58:32 PM
Subject: Re: [ovirt-users] VM failover with ovirt3.5
Hi
I had a try with "hosted-engine --set-maintence --mode=local" on
compute2-1, which is host 3 in my cluster. From the log, it shows
maintence mode is dectected, but migration does not happen.
The logs are as follows. Is there any other config I need to check?
[root@compute2-1 vdsm]# hosted-engine --vm-status
--== Host 1 status ==-
Status up-to-date : True
Hostname : 10.0.0.94
Host ID : 1
Engine status : {"health": "good", "vm": "up",
"detail": "up"}
Score : 2400
Local maintenance : False
Host timestamp : 836296
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=836296 (Sat Dec 27 11:42:39 2014)
host-id=1
score=2400
maintenance=False
state=EngineUp
--== Host 2 status ==--
Status up-to-date : True
Hostname : 10.0.0.93
Host ID : 2
Engine status : {"reason": "vm not running on
this host", "health": "bad", "vm": "down",
"detail": "unknown"}
Score : 2400
Local maintenance : False
Host timestamp : 687358
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=687358 (Sat Dec 27 08:42:04 2014)
host-id=2
score=2400
maintenance=False
state=EngineDown
--== Host 3 status ==--
Status up-to-date : True
Hostname : 10.0.0.92
Host ID : 3
Engine status : {"reason": "vm not running on
this host", "health": "bad", "vm": "down",
"detail": "unknown"}
Score : 0
Local maintenance : True
Host timestamp : 681827
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=681827 (Sat Dec 27 08:42:40 2014)
host-id=3
score=0
maintenance=True
state=LocalMaintenance
[root@compute2-1 vdsm]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log
MainThread::INFO::2014-12-27
08:42:41,109::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-27
08:42:51,198::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-27
08:42:51,420::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-27
08:42:51,420::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-27
08:43:01,507::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-27
08:43:01,773::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-27
08:43:01,773::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-27
08:43:11,859::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
Local maintenance detected
MainThread::INFO::2014-12-27
08:43:12,072::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state LocalMaintenance (score: 0)
MainThread::INFO::2014-12-27
08:43:12,072::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
[root@compute2-3 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log
MainThread::INFO::2014-12-27
11:36:28,855::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-27
11:36:39,130::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-27
11:36:39,130::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-27
11:36:49,449::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-27
11:36:49,449::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-27
11:36:59,739::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-27
11:36:59,739::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-27
11:37:09,779::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine vm running on localhost
MainThread::INFO::2014-12-27
11:37:10,026::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-27
11:37:10,026::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-27
11:37:20,331::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-27
11:37:20,331::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
[root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log
MainThread::INFO::2014-12-27
08:36:12,462::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-27
08:36:22,797::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-27
08:36:22,798::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-27
08:36:32,876::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine vm is running on host 10.0.0.94 (id 1)
MainThread::INFO::2014-12-27
08:36:33,169::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-27
08:36:33,169::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-27
08:36:43,567::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-27
08:36:43,567::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-27
08:36:53,858::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-27
08:36:53,858::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-27
08:37:04,028::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)
Global metadata: {'maintenance': False}
MainThread::INFO::2014-12-27
08:37:04,028::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)
Host 10.0.0.94 (id 1): {'extra':
'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=835987
(Sat Dec 27 11:37:30
2014)\nhost-id=1\nscore=2400\nmaintenance=False\nstate=EngineUp\n',
'hostname': '10.0.0.94', 'alive': True, 'host-id': 1,
'engine-status':
{'health': 'good', 'vm': 'up', 'detail':
'up'}, 'score': 2400,
'maintenance': False, 'host-ts': 835987}
MainThread::INFO::2014-12-27
08:37:04,028::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)
Host 10.0.0.92 (id 3): {'extra':
'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=681528
(Sat Dec 27 08:37:41
2014)\nhost-id=3\nscore=0\nmaintenance=True\nstate=LocalMaintenance\n',
'hostname': '10.0.0.92', 'alive': True, 'host-id': 3,
'engine-status':
{'reason': 'vm not running on this host', 'health': 'bad',
'vm':
'down', 'detail': 'unknown'}, 'score': 0,
'maintenance': True,
'host-ts': 681528}
MainThread::INFO::2014-12-27
08:37:04,028::state_machine::168::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)
Local (id 2): {'engine-health': {'reason': 'vm not running on this
host', 'health': 'bad', 'vm': 'down',
'detail': 'unknown'}, 'bridge':
True, 'mem-free': 15300.0, 'maintenance': False, 'cpu-load':
0.0215,
'gateway': True}
MainThread::INFO::2014-12-27
08:37:04,265::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-27
08:37:04,265::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
Thanks,
Cong
On 2014/12/22, at 5:29, "Simone Tiraboschi"
<stirabos@redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>>
wrote:
----- Original Message -----
From: "Cong Yue"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>>
To: "Simone Tiraboschi"
<stirabos@redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>>
Cc:
users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org>
Sent: Friday, December 19, 2014 7:22:10 PM
Subject: RE: [ovirt-users] VM failover with ovirt3.5
Thanks for the information. This is the log for my three ovirt nodes.
From the output of hosted-engine --vm-status, it shows the engine
state for
my 2nd and 3rd ovirt node is DOWN.
Is this the reason why VM failover not work in my environment?
No, they looks ok: you can run the engine VM on single host at a time.
How can I make
also engine works for my 2nd and 3rd ovit nodes?
If you put the host 1 in local maintenance mode ( hosted-engine --set-maintenance
--mode=local ) the VM should migrate to host 2; if you reactivate host 1 ( hosted-engine
--set-maintenance --mode=none ) and put host 2 in local maintenance mode the VM should
migrate again.
Can you please try that and post the logs if something is going bad?
--
--== Host 1 status ==--
Status up-to-date : True
Hostname : 10.0.0.94
Host ID : 1
Engine status : {"health": "good", "vm": "up",
"detail": "up"}
Score : 2400
Local maintenance : False
Host timestamp : 150475
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=150475 (Fri Dec 19 13:12:18 2014)
host-id=1
score=2400
maintenance=False
state=EngineUp
--== Host 2 status ==--
Status up-to-date : True
Hostname : 10.0.0.93
Host ID : 2
Engine status : {"reason": "vm not running on
this host", "health": "bad", "vm": "down",
"detail": "unknown"}
Score : 2400
Local maintenance : False
Host timestamp : 1572
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1572 (Fri Dec 19 10:12:18 2014)
host-id=2
score=2400
maintenance=False
state=EngineDown
--== Host 3 status ==--
Status up-to-date : False
Hostname : 10.0.0.92
Host ID : 3
Engine status : unknown stale-data
Score : 2400
Local maintenance : False
Host timestamp : 987
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=987 (Fri Dec 19 10:09:58 2014)
host-id=3
score=2400
maintenance=False
state=EngineDown
--
And the /var/log/ovirt-hosted-engine-ha/agent.log for three ovirt nodes are
as follows:
--
10.0.0.94(hosted-engine-1)
---
MainThread::INFO::2014-12-19
13:09:33,716::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:09:33,716::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:09:44,017::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:09:44,017::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:09:54,303::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:09:54,303::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:10:04,342::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine vm running on localhost
MainThread::INFO::2014-12-19
13:10:04,617::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:10:04,617::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:10:14,657::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)
Global metadata: {'maintenance': False}
MainThread::INFO::2014-12-19
13:10:14,657::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)
Host 10.0.0.93 (id 2): {'extra':
'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1448
(Fri Dec 19 10:10:14
2014)\nhost-id=2\nscore=2400\nmaintenance=False\nstate=EngineDown\n',
'hostname': '10.0.0.93', 'alive': True, 'host-id': 2,
'engine-status':
{'reason': 'vm not running on this host', 'health': 'bad',
'vm':
'down', 'detail': 'unknown'}, 'score': 2400,
'maintenance': False,
'host-ts': 1448}
MainThread::INFO::2014-12-19
13:10:14,657::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)
Host 10.0.0.92 (id 3): {'extra':
'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=987
(Fri Dec 19 10:09:58
2014)\nhost-id=3\nscore=2400\nmaintenance=False\nstate=EngineDown\n',
'hostname': '10.0.0.92', 'alive': True, 'host-id': 3,
'engine-status':
{'reason': 'vm not running on this host', 'health': 'bad',
'vm':
'down', 'detail': 'unknown'}, 'score': 2400,
'maintenance': False,
'host-ts': 987}
MainThread::INFO::2014-12-19
13:10:14,658::state_machine::168::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)
Local (id 1): {'engine-health': {'health': 'good', 'vm':
'up',
'detail': 'up'}, 'bridge': True, 'mem-free': 1079.0,
'maintenance':
False, 'cpu-load': 0.0269, 'gateway': True}
MainThread::INFO::2014-12-19
13:10:14,904::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:10:14,904::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:10:25,210::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:10:25,210::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:10:35,499::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:10:35,499::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:10:45,784::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:10:45,785::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:10:56,070::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:10:56,070::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:11:06,109::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
Engine vm running on localhost
MainThread::INFO::2014-12-19
13:11:06,359::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:11:06,359::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:11:16,658::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:11:16,658::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:11:26,991::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:11:26,991::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
MainThread::INFO::2014-12-19
13:11:37,341::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 2400)
MainThread::INFO::2014-12-19
13:11:37,341::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.93 (id: 2, score: 2400)
----
10.0.0.93 (hosted-engine-2)
MainThread::INFO::2014-12-19
10:12:18,339::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-19
10:12:18,339::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-19
10:12:28,651::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-19
10:12:28,652::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-19
10:12:39,010::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-19
10:12:39,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-19
10:12:49,338::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-19
10:12:49,338::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-19
10:12:59,642::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-19
10:12:59,642::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
MainThread::INFO::2014-12-19
10:13:10,010::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 2400)
MainThread::INFO::2014-12-19
10:13:10,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host 10.0.0.94 (id: 1, score: 2400)
10.0.0.92(hosted-engine-3)
same as 10.0.0.93
--
-----Original Message-----
From: Simone Tiraboschi [mailto:stirabos@redhat.com]
Sent: Friday, December 19, 2014 12:28 AM
To: Yue, Cong
Cc:
users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org>
Subject: Re: [ovirt-users] VM failover with ovirt3.5
----- Original Message -----
From: "Cong Yue"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>>
To:
users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org>
Sent: Friday, December 19, 2014 2:14:33 AM
Subject: [ovirt-users] VM failover with ovirt3.5
Hi
In my environment, I have 3 ovirt nodes as one cluster. And on top of
host-1, there is one vm to host ovirt engine.
Also I have one external storage for the cluster to use as data domain
of engine and data.
I confirmed live migration works well in my environment.
But it seems very buggy for VM failover if I try to force to shut down
one ovirt node. Sometimes the VM in the node which is shutdown can
migrate to other host, but it take more than several minutes.
Sometimes, it can not migrate at all. Sometimes, only when the host is
back, the VM is beginning to move.
Can you please check or share the logs under /var/log/ovirt-hosted-engine-ha/
?
Is there some documentation to explain how VM failover is working? And
is there some bugs reported related with this?
http://www.ovirt.org/Features/Self_Hosted_Engine#Agent_State_Diagram
Thanks in advance,
Cong
This e-mail message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any
unauthorized review, use, disclosure or distribution is prohibited. If
you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message. If you are the
intended recipient, please be advised that the content of this message
is subject to access, review and disclosure by the sender's e-mail System
Administrator.
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org><mailto:Users@ovirt.org><mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
This e-mail message is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. Any unauthorized review,
use, disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy all copies
of the original message. If you are the intended recipient, please be
advised that the content of this message is subject to access, review and
disclosure by the sender's e-mail System Administrator.
This e-mail message is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient, please contact the
sender by reply e-mail and destroy all copies of the original message. If you are the
intended recipient, please be advised that the content of this message is subject to
access, review and disclosure by the sender's e-mail System Administrator.
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org><mailto:Users@ovirt.org><mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
________________________________
This e-mail message is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient, please contact the
sender by reply e-mail and destroy all copies of the original message. If you are the
intended recipient, please be advised that the content of this message is subject to
access, review and disclosure by the sender's e-mail System Administrator.
________________________________
This e-mail message is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient, please contact the
sender by reply e-mail and destroy all copies of the original message. If you are the
intended recipient, please be advised that the content of this message is subject to
access, review and disclosure by the sender's e-mail System Administrator.
________________________________
This e-mail message is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient, please contact the
sender by reply e-mail and destroy all copies of the original message. If you are the
intended recipient, please be advised that the content of this message is subject to
access, review and disclosure by the sender's e-mail System Administrator.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.ovirt.org/pipermail/users/attachments/20141229/4ec6cc13/atta...
------------------------------
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
End of Users Digest, Vol 39, Issue 171
**************************************
------=_Part_1882534_1428653136.1419879250032
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: georgia,serif; font-size: 12pt;
colo=
r: #000000"><div>Hi,</div><div>Can you please provide engine.log
from =
/var/log/ovirt-engine/engine.log and to try as
follows:</div><div><ol><li>R=
evert all tree hosts to maintenance-mode=3Dnone.</li><li>Check that engine =
up and running.</li><li>Turn one of the hosts that is not running the engin=
e, to maintenance mode local.</li><li>Turn host that is running the engine =
to maintenance mode local.</li><li>Check that engine migrated to one and th=
e only remaining host, that had not been put in to maintenance mode at all.=
</li></ol><div>Can you also provide your engine version, is it 3.4
somethin=
g?</div></div><div><br></div><div><span
name=3D"x"></span><br>Thanks in adv=
ance.<br><div><br></div>Best
regards,<br>Nikolai<br>____________________<br=
Nikolai Sednev<br>Senior Quality Engineer at Compute
team<br>Red Hat Israe=
l<br>34 Jerusalem Road,<br>Ra'anana,
Israel 43501<br><div><br></div>Tel: &n=
bsp; +972 9 7692043<br>Mobile: +972 52
7342734<br>Emai=
l: nsednev(a)redhat.com<br>IRC: nsednev<span
name=3D"x"></span><br></div><div=
<br></div><hr id=3D"zwchr"><div
style=3D"color:#000;font-weight:normal;fon=
t-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;=
font-size:12pt;"><b>From:
</b>users-request(a)ovirt.org<br><b>To: </b>users@o=
virt.org<br><b>Sent: </b>Monday, December 29, 2014 8:29:36
PM<br><b>Subject=
: </b>Users Digest, Vol 39, Issue 171<br><div><br></div>Send
Users mailing =
list submissions
to<br> user=
s(a)ovirt.org<br><div><br></div>To subscribe or unsubscribe via the
World Wid=
e Web,
visit<br> http://list=
s.ovirt.org/mailman/listinfo/users<br>or, via email, send a message with su=
bject or body 'help'
to<br> =
users-request(a)ovirt.org<br><div><br></div>You can reach the person
managing=
the list
at<br> users-owner=
@ovirt.org<br><div><br></div>When replying, please edit your
Subject line s=
o it is more specific<br>than "Re: Contents of Users
digest..."<br><div><br=
</div><br>Today's
Topics:<br><div><br></div> 1. Re: VM
f=
ailover with ovirt3.5 (Yue,
Cong)<br><div><br></div><br>-------------------=
---------------------------------------------------<br><div><br></div>Messa=
ge: 1<br>Date: Mon, 29 Dec 2014 10:29:04 -0800<br>From: "Yue, Cong"
<Con=
g_Yue(a)alliedtelesis.com&gt;<br>To: Artyom Lukianov
&lt;alukiano(a)redhat.com&=
gt;<br>Cc: "users(a)ovirt.org"
&lt;users(a)ovirt.org&gt;<br>Subject: Re: [ovirt=
-users] VM failover with ovirt3.5<br>Message-ID: <21D302CF-AD6F-4E8C-A37=
3-52ADAC1C129B(a)alliedtelesis.com&gt;<br>Content-Type: text/plain; charset=
=3D"utf-8"<br><div><br></div>I disabled local
maintenance mode for all host=
s, and then only set the host where HE VM is there to local maintenance mod=
e. The logs are as follows. During the migration of HE VM , it shows some f=
atal error happen. By the way, also HE VM can not work with live migration.=
Instead, other VMs can do live
migration.<br><div><br></div>---<br>[root@c=
ompute2-3 ~]# hosted-engine --set-maintenance --mode=3Dlocal<br>You have ne=
w mail in /var/spool/mail/root<br>[root@compute2-3 ~]# tail -f /var/log/ovi=
rt-hosted-engine-ha/agent.log<br>MainThread::INFO::2014-12-29<br>13:16:12,4=
35::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEn=
gine::(start_monitoring)<br>Best remote host 10.0.0.92 (id: 3, score: 2400)=
<br>MainThread::INFO::2014-12-29<br>13:16:22,711::hosted_engine::327::ovirt=
_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>C=
urrent state EngineUp (score: 2400)<br>MainThread::INFO::2014-12-29<br>13:1=
6:22,711::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Ho=
stedEngine::(start_monitoring)<br>Best remote host 10.0.0.92 (id: 3, score:=
2400)<br>MainThread::INFO::2014-12-29<br>13:16:32,978::hosted_engine::327:=
:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring=
)<br>Current state EngineUp (score:
2400)<br>MainThread::INFO::2014-12-29<b=
r>13:16:32,978::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_eng=
ine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2, =
score: 2400)<br>MainThread::INFO::2014-12-29<br>13:16:43,272::hosted_engine=
::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni=
toring)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::2014-1=
2-29<br>13:16:43,272::hosted_engine::332::ovirt_hosted_engine_ha.agent.host=
ed_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (i=
d: 2, score: 2400)<br>MainThread::INFO::2014-12-29<br>13:16:53,316::states:=
:394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)<br=
Engine vm running on
localhost<br>MainThread::INFO::2014-12-29<br>13:16:53=
,562::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted=
Engine::(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainT=
hread::INFO::2014-12-29<br>13:16:53,562::hosted_engine::332::ovirt_hosted_e=
ngine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remot=
e host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-29<br>13=
:17:03,600::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engi=
ne.HostedEngine::(check)<br>Local maintenance detected<br>MainThread::INFO:=
:2014-12-29<br>13:17:03,611::brokerlink::111::ovirt_hosted_engine_ha.lib.br=
okerlink.BrokerLink::(notify)<br>Trying: notify time=3D1419877023.61 type=
=3Dstate_transition<br>detail=3DEngineUp-LocalMaintenanceMigrateVm hostname=
=3D'compute2-3'<br>MainThread::INFO::2014-12-29<br>13:17:03,672::brokerlink=
::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)<br>Succes=
s, was notification of state_transition<br>(EngineUp-LocalMaintenanceMigrat=
eVm) sent? sent<br>MainThread::INFO::2014-12-29<br>13:17:03,911::states::20=
8::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score)<br>Scor=
e is 0 due to local maintenance mode<br>MainThread::INFO::2014-12-29<br>13:=
17:03,912::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.H=
ostedEngine::(start_monitoring)<br>Current state LocalMaintenanceMigrateVm =
(score: 0)<br>MainThread::INFO::2014-12-29<br>13:17:03,912::hosted_engine::=
332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monito=
ring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INF=
O::2014-12-29<br>13:17:03,960::brokerlink::111::ovirt_hosted_engine_ha.lib.=
brokerlink.BrokerLink::(notify)<br>Trying: notify time=3D1419877023.96 type=
=3Dstate_transition<br>detail=3DLocalMaintenanceMigrateVm-EngineMigratingAw=
ay<br>hostname=3D'compute2-3'<br>MainThread::INFO::2014-12-29<br>13:17:03,9=
80::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(not=
ify)<br>Success, was notification of state_transition<br>(LocalMaintenanceM=
igrateVm-EngineMigratingAway) sent? sent<br>MainThread::INFO::2014-12-29<br=
13:17:04,218::states::66::ovirt_hosted_engine_ha.agent.hosted_engine.Hoste=
dEngine::(_penalize_memory)<br>Penalizing score by 400 due to low free memo=
ry<br>MainThread::INFO::2014-12-29<br>13:17:04,218::hosted_engine::327::ovi=
rt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br=
Current state EngineMigratingAway (score:
2000)<br>MainThread::INFO::2014-=
12-29<br>13:17:04,219::hosted_engine::332::ovirt_hosted_engine_ha.agent.hos=
ted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (=
id: 2, score: 2400)<br>MainThread::ERROR::2014-12-29<br>13:17:14,251::hoste=
d_engine::867::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_m=
onitor_migration)<br>Failed to migrate<br>Traceback (most recent call last)=
:<br> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/ag=
ent/hosted_engine.py",<br>line 863, in
_monitor_migration<br> v=
m_id,<br> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_h=
a/lib/vds_client.py",<br>line 85, in
run_vds_client_cmd<br> res=
ponse['status']['message'])<br>DetailedError: Error 12 from
migrateStatus: =
Fatal error during migration<br>MainThread::INFO::2014-12-29<br>13:17:14,26=
2::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(noti=
fy)<br>Trying: notify time=3D1419877034.26 type=3Dstate_transition<br>detai=
l=3DEngineMigratingAway-ReinitializeFSM
hostname=3D'compute2-3'<br>MainThre=
ad::INFO::2014-12-29<br>13:17:14,263::brokerlink::120::ovirt_hosted_engine_=
ha.lib.brokerlink.BrokerLink::(notify)<br>Success, was notification of stat=
e_transition<br>(EngineMigratingAway-ReinitializeFSM) sent? ignored<br>Main=
Thread::INFO::2014-12-29<br>13:17:14,496::hosted_engine::327::ovirt_hosted_=
engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current s=
tate ReinitializeFSM (score: 0)<br>MainThread::INFO::2014-12-29<br>13:17:14=
,496::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted=
Engine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2, score: 240=
0)<br>MainThread::INFO::2014-12-29<br>13:17:24,536::state_decorators::124::=
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>Local m=
aintenance detected<br>MainThread::INFO::2014-12-29<br>13:17:24,547::broker=
link::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)<br>Tr=
ying: notify time=3D1419877044.55 type=3Dstate_transition<br>detail=3DReini=
tializeFSM-LocalMaintenance
hostname=3D'compute2-3'<br>MainThread::INFO::20=
14-12-29<br>13:17:24,574::brokerlink::120::ovirt_hosted_engine_ha.lib.broke=
rlink.BrokerLink::(notify)<br>Success, was notification of state_transition=
<br>(ReinitializeFSM-LocalMaintenance) sent? sent<br>MainThread::INFO::2014=
-12-29<br>13:17:24,812::hosted_engine::327::ovirt_hosted_engine_ha.agent.ho=
sted_engine.HostedEngine::(start_monitoring)<br>Current state LocalMaintena=
nce (score: 0)<br>MainThread::INFO::2014-12-29<br>13:17:24,812::hosted_engi=
ne::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_mo=
nitoring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread:=
:INFO::2014-12-29<br>13:17:34,851::state_decorators::124::ovirt_hosted_engi=
ne_ha.agent.hosted_engine.HostedEngine::(check)<br>Local maintenance detect=
ed<br>MainThread::INFO::2014-12-29<br>13:17:35,095::hosted_engine::327::ovi=
rt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br=
Current state LocalMaintenance (score:
0)<br>MainThread::INFO::2014-12-29<=
br>13:17:35,095::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_en=
gine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2,=
score: 2400)<br>MainThread::INFO::2014-12-29<br>13:17:45,130::state_decora=
tors::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)=
<br>Local maintenance
detected<br>MainThread::INFO::2014-12-29<br>13:17:45,=
368::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE=
ngine::(start_monitoring)<br>Current state LocalMaintenance (score: 0)<br>M=
ainThread::INFO::2014-12-29<br>13:17:45,368::hosted_engine::332::ovirt_host=
ed_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best r=
emote host 10.0.0.93 (id: 2, score: 2400)<br>^C<br>[root@compute2-3
~]#<br>=
<div><br></div><br>[root@compute2-3 ~]# hosted-engine
--vm-status<br><div><=
br></div><br>--=3D=3D Host 1 status
=3D=3D--<br><div><br></div>Status up-to=
-date
: True<=
br>Hostname
=
: 10.0.0.94<br>Host ID
&nb=
sp;
: =
1<br>Engine status
=
: {"health": "good", "vm":
"up",<br>"detail": "up"}<br>=
Score
=
: 0<br>Local maintenance
&=
nbsp; :
True<br>Host timestamp &nb=
sp;
: 101495=
6<tel:1014956><br>Extra metadata (valid at
timestamp):<br>metadata_pa=
rse_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D1014956<t=
el:1014956> (Mon Dec 29 13:20:19
2014)<br>host-id=3D1<br>score=3D0<br>ma=
intenance=3DTrue<br>state=3DLocalMaintenance<br><div><br></div><br>--=3D=3D=
Host 2 status =3D=3D--<br><div><br></div>Status up-to-date
&=
nbsp; :
True<br>Hostname &n=
bsp;
=
: 10.0.0.93<br>Host ID
&nb=
sp; :
2<br>Engine status &n=
bsp;
:=
{"reason": "vm not running on<br>this host",
"health": "bad", "vm": "down"=
, "detail": "unknown"}<br>Score
&=
nbsp;
: 2400<br>Loca=
l maintenance
 =
;: False<br>Host timestamp
=
: 866019<br>Extra metadata (valid at
timestamp):<br>m=
etadata_parse_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D86=
6019 (Mon Dec 29 10:19:45
2014)<br>host-id=3D2<br>score=3D2400<br>maintenan=
ce=3DFalse<br>state=3DEngineDown<br><div><br></div><br>--=3D=3D
Host 3 stat=
us =3D=3D--<br><div><br></div>Status up-to-date
=
: True<br>Hostname
=
: 10.=
0.0.92<br>Host ID
&=
nbsp; : 3<br>Engine status
=
: {"reason": =
"vm not running on<br>this host", "health": "bad",
"vm": "down", "detail": =
"unknown"}<br>Score
=
:
2400<br>Local maintenanc=
e
: False<br>=
Host timestamp
&nbs=
p; : 860493<br>Extra metadata (valid at
timestamp):<br>metadata_pars=
e_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D860493 (Mon De=
c 29 10:20:35
2014)<br>host-id=3D3<br>score=3D2400<br>maintenance=3DFalse<b=
r>state=3DEngineDown<br>[root@compute2-3
~]#<br>---<br>Thanks,<br>Cong<br><=
div><br></div><br><div><br></div>On 2014/12/29, at
8:43, "Artyom Lukianov" =
<alukiano@redhat.com<mailto:alukiano@redhat.com>>
wrote:<br><di=
v><br></div>I see that HE vm run on host with ip 10.0.0.94, and two
another=
hosts in "Local Maintenance" state, so vm will not migrate to any of them,=
can you try disable local maintenance on all hosts in HE environment and a=
fter enable "local maintenance" on host where HE vm run, and provide also o=
utput of hosted-engine --vm-status.<br>Failover works in next way:<br>1) if=
host where run HE vm have score less by 800 that some other host in HE env=
ironment, HE vm will migrate on host with best score<br>2) if something hap=
pen to vm(kernel panic, crash of service...), agent will restart HE vm on a=
nother host in HE environment with positive score<br>3) if put to local mai=
ntenance host with HE vm, vm will migrate to another host with positive sco=
re<br>Thanks.<br><div><br></div>----- Original Message
-----<br>From: "Cong=
Yue"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com&g=
t;><br>To: "Artyom Lukianov"
<alukiano@redhat.com<mailto:alukiano@=
redhat.com>><br>Cc: "Simone Tiraboschi"
&lt;stirabos(a)redhat.com&lt;ma=
ilto:stirabos@redhat.com>>, users@ovirt.org<mailto:users@ovirt.org=
><br>Sent: Monday, December 29, 2014 6:30:42 PM<br>Subject: Re:
[ovirt-u=
sers] VM failover with ovirt3.5<br><div><br></div>Thanks and the
--vm-statu=
s log is as follows:<br>[root@compute2-2 ~]# hosted-engine
--vm-status<br><=
div><br></div><br>--=3D=3D Host 1 status
=3D=3D--<br><div><br></div>Status =
up-to-date
: =
True<br>Hostname
&n=
bsp; : 10.0.0.94<br>Host ID
 =
;
&nb=
sp;: 1<br>Engine status
&n=
bsp; : {"health": "good",
"vm": "up",<br>"detail": "up"=
}<br>Score
&=
nbsp; : 2400<br>Local
maintenance =
:
False<br>Host time=
stamp
=
: 1008087<br>Extra metadata (valid at timestamp):<br>metadata_parse_versio=
n=3D1<br>metadata_feature_version=3D1<br>timestamp=3D1008087<tel:1008087=
> (Mon Dec 29 11:25:51
2014)<br>host-id=3D1<br>score=3D2400<br>maintenan=
ce=3DFalse<br>state=3DEngineUp<br><div><br></div><br>--=3D=3D
Host 2 status=
=3D=3D--<br><div><br></div>Status up-to-date
&=
nbsp; : True<br>Hostname
&n=
bsp;
: 10.0.=
0.93<br>Host ID
&nb=
sp; : 2<br>Engine status
&n=
bsp;
: {"reason": "v=
m not running on<br>this host", "health": "bad",
"vm": "down", "detail": "u=
nknown"}<br>Score
&=
nbsp; : 0<br>Local
maintenance &nb=
sp;
: True<br>Host t=
imestamp
&nb=
sp; : 859142<br>Extra metadata (valid at timestamp):<br>metadata_parse_vers=
ion=3D1<br>metadata_feature_version=3D1<br>timestamp=3D859142 (Mon Dec 29 0=
8:25:08
2014)<br>host-id=3D2<br>score=3D0<br>maintenance=3DTrue<br>state=3D=
LocalMaintenance<br><div><br></div><br>--=3D=3D Host 3
status =3D=3D--<br><=
div><br></div>Status up-to-date
&=
nbsp; : True<br>Hostname
&n=
bsp; :
10.0.0.92<br>Host I=
D
&nb=
sp; : 3<br>Engine status
&n=
bsp; :
{"reason": "vm not running =
on<br>this host", "health": "bad", "vm":
"down", "detail": "unknown"}<br>Sc=
ore
&=
nbsp; : 0<br>Local maintenance
&nb=
sp; : True<br>Host
timestamp  =
;
: 853615<b=
r>Extra metadata (valid at
timestamp):<br>metadata_parse_version=3D1<br>met=
adata_feature_version=3D1<br>timestamp=3D853615 (Mon Dec 29 08:25:57 2014)<=
br>host-id=3D3<br>score=3D0<br>maintenance=3DTrue<br>state=3DLocalMaintenan=
ce<br>You have new mail in /var/spool/mail/root<br>[root@compute2-2
~]#<br>=
<div><br></div>Could you please explain how VM failover works inside
ovirt?=
Is there any other debug option I can enable to check the problem?<br><div=
<br></div>Thanks,<br>Cong<br><div><br></div><br>On
2014/12/29, at 1:39, "A=
rtyom Lukianov"
<alukiano@redhat.com<mailto:alukiano@redhat.com>&l=
t;mailto:alukiano@redhat.com>>
wrote:<br><div><br></div>Can you also =
provide output of hosted-engine --vm-status please, previous time it was us=
eful, because I do not see something
unusual.<br>Thanks<br><div><br></div>-=
---- Original Message -----<br>From: "Cong Yue"
&lt;Cong_Yue(a)alliedtelesis.=
com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedteles=
is.com>><br>To: "Artyom Lukianov"
<alukiano@redhat.com<mailto:a=
lukiano@redhat.com><mailto:alukiano@redhat.com>><br>Cc:
"Simone=
Tiraboschi"
<stirabos@redhat.com<mailto:stirabos@redhat.com><m=
ailto:stirabos@redhat.com>>, users@ovirt.org<mailto:users@ovirt.or=
g><mailto:users@ovirt.org><br>Sent: Monday, December 29, 2014
7:15=
:24 AM<br>Subject: Re: [ovirt-users] VM failover with
ovirt3.5<br><div><br>=
</div>Also I change the maintenance mode to local in another host. But also=
the VM in this host can not be migrated. The logs are as follows.<br><div>=
<br></div>[root@compute2-2 ~]# hosted-engine --set-maintenance --mode=3Dloc=
al<br>[root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.lo=
g<br>MainThread::INFO::2014-12-28<br>21:09:04,184::hosted_engine::332::ovir=
t_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>=
Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-1=
2-28<br>21:09:14,603::hosted_engine::327::ovirt_hosted_engine_ha.agent.host=
ed_engine.HostedEngine::(start_monitoring)<br>Current state EngineDown (sco=
re: 2400)<br>MainThread::INFO::2014-12-28<br>21:09:14,603::hosted_engine::3=
32::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitor=
ing)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO=
::2014-12-28<br>21:09:24,903::hosted_engine::327::ovirt_hosted_engine_ha.ag=
ent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state EngineD=
own (score: 2400)<br>MainThread::INFO::2014-12-28<br>21:09:24,904::hosted_e=
ngine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start=
_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThre=
ad::INFO::2014-12-28<br>21:09:35,026::states::437::ovirt_hosted_engine_ha.a=
gent.hosted_engine.HostedEngine::(consume)<br>Engine vm is running on host =
10.0.0.94 (id 1)<br>MainThread::INFO::2014-12-28<br>21:09:35,236::hosted_en=
gine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_=
monitoring)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::=
2014-12-28<br>21:09:35,236::hosted_engine::332::ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0=
.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-28<br>21:09:45,604::h=
osted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:=
:(start_monitoring)<br>Current state EngineDown (score: 2400)<br>MainThread=
::INFO::2014-12-28<br>21:09:45,604::hosted_engine::332::ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote hos=
t 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-28<br>21:09:5=
5,691::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.Ho=
stedEngine::(check)<br>Local maintenance detected<br>MainThread::INFO::2014=
-12-28<br>21:09:55,701::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerl=
ink.BrokerLink::(notify)<br>Trying: notify time=3D1419829795.7 type=3Dstate=
_transition<br>detail=3DEngineDown-LocalMaintenance
hostname=3D'compute2-2'=
<br>MainThread::INFO::2014-12-28<br>21:09:55,761::brokerlink::120::ovirt_ho=
sted_engine_ha.lib.brokerlink.BrokerLink::(notify)<br>Success, was notifica=
tion of state_transition<br>(EngineDown-LocalMaintenance) sent? sent<br>Mai=
nThread::INFO::2014-12-28<br>21:09:55,990::states::208::ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine::(score)<br>Score is 0 due to local ma=
intenance mode<br>MainThread::INFO::2014-12-28<br>21:09:55,990::hosted_engi=
ne::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_mo=
nitoring)<br>Current state LocalMaintenance (score: 0)<br>MainThread::INFO:=
:2014-12-28<br>21:09:55,991::hosted_engine::332::ovirt_hosted_engine_ha.age=
nt.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.=
0.94 (id: 1, score: 2400)<br>^C<br>You have new mail in /var/spool/mail/roo=
t<br>[root@compute2-2 ~]# ps -ef | grep qemu<br>root
18420 &n=
bsp;2777 0 21:10<x-apple-data-detectors://39> pts/0
&nbs=
p;00:00:00<x-apple-data-detectors://40> grep --color=3Dauto
qemu<br>q=
emu 29809 1 0 Dec19 ?
 =
; 01:17:20 /usr/libexec/qemu-kvm<br>-name testvm2-2 -S -machine rhel6=
.5.0,accel=3Dkvm,usb=3Doff -cpu Nehalem<br>-m 500 -realtime mlock=3Doff -sm=
p<br>1,maxcpus=3D16,sockets=3D16,cores=3D1,threads=3D1 -uuid<br>c31e97d0-13=
5e-42da-9954-162b5228dce3 -smbios<br>type=3D1,manufacturer=3DoVirt,product=
=3DoVirt<br>Node,version=3D7-0.1406.el7.centos.2.5,serial=3D4C4C4544-0059-3=
610-8033-B4C04F395931,uuid=3Dc31e97d0-135e-42da-9954-162b5228dce3<br>-no-us=
er-config -nodefaults -chardev<br>socket,id=3Dcharmonitor,path=3D/var/lib/l=
ibvirt/qemu/testvm2-2.monitor,server,nowait<br>-mon chardev=3Dcharmonitor,i=
d=3Dmonitor,mode=3Dcontrol -rtc<br>base=3D2014-12-19T20:17:17<x-apple-da=
ta-detectors://42>,driftfix=3Dslew -no-kvm-pit-reinjection<br>-no-hpet -=
no-shutdown -boot strict=3Don -device<br>piix3-usb-uhci,id=3Dusb,bus=3Dpci.=
0,addr=3D0x1.0x2 -device<br>virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0=
x4 -device<br>virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpc=
i.0,addr=3D0x5<br>-drive if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don,format=
=3Draw,serial=3D<br>-device ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-=
1-0,id=3Dide0-1-0<br>-drive file=3D/rhev/data-center/00000002-0002-0002-000=
2-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/5cbeb8c9-4f04-48=
d0-a5eb-78c49187c550/a0570e8c-9867-4ec4-818f-11e102fc4f9b,if=3Dnone,id=3Ddr=
ive-virtio-disk0,format=3Dqcow2,serial=3D5cbeb8c9-4f04-48d0-a5eb-78c49187c5=
50,cache=3Dnone,werror=3Dstop,rerror=3Dstop,aio=3Dthreads<br>-device virtio=
-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk0,id=3D=
virtio-disk0,bootindex=3D1<br>-netdev tap,fd=3D28,id=3Dhostnet0,vhost=3Don,=
vhostfd=3D29 -device<br>virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00=
:1a:4a:db:94:00,bus=3Dpci.0,addr=3D0x3<br>-chardev socket,id=3Dcharchannel0=
,path=3D/var/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3=
.com.redhat.rhevm.vdsm,server,nowait<br>-device virtserialport,bus=3Dvirtio=
-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dchannel0,name=3Dcom.redhat.rh=
evm.vdsm<br>-chardev socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/=
channels/c31e97d0-135e-42da-9954-162b5228dce3.org.qemu.guest_agent.0,server=
,nowait<br>-device virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dc=
harchannel1,id=3Dchannel1,name=3Dorg.qemu.guest_agent.0<br>-chardev spicevm=
c,id=3Dcharchannel2,name=3Dvdagent -device<br>virtserialport,bus=3Dvirtio-s=
erial0.0,nr=3D3,chardev=3Dcharchannel2,id=3Dchannel2,name=3Dcom.redhat.spic=
e.0<br>-spice tls-port=3D5901,addr=3D10.0.0.93,x509-dir=3D/etc/pki/vdsm/lib=
virt-spice,tls-channel=3Dmain,tls-channel=3Ddisplay,tls-channel=3Dinputs,tl=
s-channel=3Dcursor,tls-channel=3Dplayback,tls-channel=3Drecord,tls-channel=
=3Dsmartcard,tls-channel=3Dusbredir,seamless-migration=3Don<br>-k en-us -vg=
a qxl -global qxl-vga.ram_size=3D67108864<tel:67108864>
-global<br>qx=
l-vga.vram_size=3D33554432<tel:33554432> -incoming tcp:[::]:49152 -de=
vice<br>virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7<br>[root@co=
mpute2-2
~]#<br><div><br></div>Thanks,<br>Cong<br><div><br></div><br>On
201=
4/12/28, at 20:53, "Yue, Cong"
<Cong_Yue@alliedtelesis.com<mailto:Con=
g_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mai=
lto:Cong_Yue@alliedtelesis.com>>
wrote:<br><div><br></div>I checked i=
t again and confirmed there is one guest VM is running on the top of this h=
ost. The log is as follows:<br><div><br></div>[root@compute2-1
vdsm]# ps -e=
f | grep qemu<br>qemu 2983 846
0 Dec19 ? &=
nbsp; 00:00:00<x-apple-data-detectors://0>
[super=
vdsmServer] <defunct><br>root 5489
3053 &nb=
sp;0 20:49<x-apple-data-detectors://1> pts/0
00:00:00<=
;x-apple-data-detectors://2> grep --color=3Dauto qemu<br>qemu
&nb=
sp; 26128 1 0 Dec19 ?
01:09:=
19 /usr/libexec/qemu-kvm<br>-name testvm2 -S -machine rhel6.5.0,accel=3Dkvm=
,usb=3Doff -cpu Nehalem -m<br>500 -realtime mlock=3Doff -smp 1,maxcpus=3D16=
,sockets=3D16,cores=3D1,threads=3D1<br>-uuid e46bca87-4df5-4287-844b-90a26f=
ccef33 -smbios<br>type=3D1,manufacturer=3DoVirt,product=3DoVirt<br>Node,ver=
sion=3D7-0.1406.el7.centos.2.5,serial=3D4C4C4544-0030-3310-8059-B8C04F58523=
1,uuid=3De46bca87-4df5-4287-844b-90a26fccef33<br>-no-user-config -nodefault=
s -chardev<br>socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/testvm2.=
monitor,server,nowait<br>-mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcon=
trol
-rtc<br>base=3D2014-12-19T20:18:01<x-apple-data-detectors://4>,d=
riftfix=3Dslew -no-kvm-pit-reinjection<br>-no-hpet -no-shutdown -boot stric=
t=3Don -device<br>piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -devic=
e<br>virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4 -device<br>virtio-se=
rial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x5<br>-driv=
e if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don,format=3Draw,serial=3D<br>-de=
vice ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0<br>-d=
rive file=3D/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096=
-27c4-4256-b2ac-bd7265525c69/images/b4b5426b-95e3-41af-b286-da245891cdaf/0f=
688d49-97e3-4f1d-84d4-ac1432d903b3,if=3Dnone,id=3Ddrive-virtio-disk0,format=
=3Dqcow2,serial=3Db4b5426b-95e3-41af-b286-da245891cdaf,cache=3Dnone,werror=
=3Dstop,rerror=3Dstop,aio=3Dthreads<br>-device virtio-blk-pci,scsi=3Doff,bu=
s=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk0,id=3Dvirtio-disk0,bootindex=
=3D1<br>-netdev tap,fd=3D26,id=3Dhostnet0,vhost=3Don,vhostfd=3D27 -device<b=
r>virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:db:94:01,bus=3D=
pci.0,addr=3D0x3<br>-chardev socket,id=3Dcharchannel0,path=3D/var/lib/libvi=
rt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef33.com.redhat.rhevm.vdsm=
,server,nowait<br>-device virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,char=
dev=3Dcharchannel0,id=3Dchannel0,name=3Dcom.redhat.rhevm.vdsm<br>-chardev s=
ocket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/e46bca87-4df5=
-4287-844b-90a26fccef33.org.qemu.guest_agent.0,server,nowait<br>-device vir=
tserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dchann=
el1,name=3Dorg.qemu.guest_agent.0<br>-chardev spicevmc,id=3Dcharchannel2,na=
me=3Dvdagent -device<br>virtserialport,bus=3Dvirtio-serial0.0,nr=3D3,charde=
v=3Dcharchannel2,id=3Dchannel2,name=3Dcom.redhat.spice.0<br>-spice tls-port=
=3D5900,addr=3D10.0.0.92,x509-dir=3D/etc/pki/vdsm/libvirt-spice,tls-channel=
=3Dmain,tls-channel=3Ddisplay,tls-channel=3Dinputs,tls-channel=3Dcursor,tls=
-channel=3Dplayback,tls-channel=3Drecord,tls-channel=3Dsmartcard,tls-channe=
l=3Dusbredir,seamless-migration=3Don<br>-k en-us -vga qxl -global qxl-vga.r=
am_size=3D67108864<tel:67108864>
-global<br>qxl-vga.vram_size=3D33554=
432<tel:33554432> -incoming tcp:[::]:49152
-device<br>virtio-balloon-=
pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7<br>[root@compute2-1 vdsm]# tail -f=
/var/log/ovirt-hosted-engine-ha/agent.log<br>MainThread::INFO::2014-12-28<=
br>20:49:27,315::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted=
_engine.HostedEngine::(check)<br>Local maintenance detected<br>MainThread::=
INFO::2014-12-28<br>20:49:27,646::hosted_engine::327::ovirt_hosted_engine_h=
a.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state Loc=
alMaintenance (score: 0)<br>MainThread::INFO::2014-12-28<br>20:49:27,646::h=
osted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:=
:(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>M=
ainThread::INFO::2014-12-28<br>20:49:37,732::state_decorators::124::ovirt_h=
osted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>Local maintena=
nce detected<br>MainThread::INFO::2014-12-28<br>20:49:37,961::hosted_engine=
::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni=
toring)<br>Current state LocalMaintenance (score: 0)<br>MainThread::INFO::2=
014-12-28<br>20:49:37,961::hosted_engine::332::ovirt_hosted_engine_ha.agent=
.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.=
94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-28<br>20:49:48,048::st=
ate_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin=
e::(check)<br>Local maintenance
detected<br>MainThread::INFO::2014-12-28<br=
20:49:48,319::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.Host=
edEngine::(score)<br>Score is 0 due to local maintenance
mode<br>MainThread=
::INFO::2014-12-28<br>20:49:48,319::hosted_engine::327::ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state L=
ocalMaintenance (score: 0)<br>MainThread::INFO::2014-12-28<br>20:49:48,319:=
:hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin=
e::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br=
<div><br></div>Thanks,<br>Cong<br><div><br></div><br>On
2014/12/28, at 3:4=
6, "Artyom Lukianov"
<alukiano@redhat.com<mailto:alukiano@redhat.com&=
gt;<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>>=
wrote:<br><div><br></div>I see that you set local maintenance on
host3 tha=
t do not have engine vm on it, so it nothing to migrate from this host.<br>=
If you set local maintenance on host1, vm must migrate to another host with=
positive score.<br>Thanks<br><div><br></div>----- Original
Message -----<b=
r>From: "Cong Yue"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@allied=
telesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue=
@alliedtelesis.com>><br>To: "Simone Tiraboschi"
&lt;stirabos(a)redhat.c=
om<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><m=
ailto:stirabos@redhat.com>><br>Cc:
users@ovirt.org<mailto:users@ov=
irt.org><mailto:users@ovirt.org><mailto:users@ovirt.org><br>=
Sent: Saturday, December 27, 2014 6:58:32 PM<br>Subject: Re: [ovirt-users] =
VM failover with
ovirt3.5<br><div><br></div>Hi<br><div><br></div>I
had a tr=
y with "hosted-engine --set-maintence --mode=3Dlocal" on<br>compute2-1,
whi=
ch is host 3 in my cluster. From the log, it shows<br>maintence mode is dec=
tected, but migration does not happen.<br><div><br></div>The logs
are as fo=
llows. Is there any other config I need to
check?<br><div><br></div>[root@c=
ompute2-1 vdsm]# hosted-engine
--vm-status<br><div><br></div><br>--=3D=3D H=
ost 1 status =3D=3D-<br><div><br></div>Status up-to-date
&nbs=
p; :
True<br>Hostname  =
;
&nb=
sp; : 10.0.0.94<br>Host ID
=
:
1<br>Engine status  =
;
: {"=
health": "good", "vm":
"up",<br>"detail": "up"}<br>Score
&nbs=
p;
&n=
bsp; : 2400<br>Local maintenance
&=
nbsp; : False<br>Host timestamp
&n=
bsp; :
836296<br>Extra metadata (=
valid at timestamp):<br>metadata_parse_version=3D1<br>metadata_feature_vers=
ion=3D1<br>timestamp=3D836296 (Sat Dec 27 11:42:39
2014)<br>host-id=3D1<br>=
score=3D2400<br>maintenance=3DFalse<br>state=3DEngineUp<br><div><br></div><=
br>--=3D=3D Host 2 status =3D=3D--<br><div><br></div>Status
up-to-date &nbs=
p;
: True<br>Hostnam=
e
&nb=
sp; : 10.0.0.93<br>Host ID
=
: 2<br>Engin=
e status
&nb=
sp; : {"reason": "vm not running on<br>this host",
"health": "bad", "=
vm": "down", "detail": "unknown"}<br>Score
&nbs=
p;
: 2=
400<br>Local maintenance
&=
nbsp; : False<br>Host timestamp
&n=
bsp; : 687358<br>Extra metadata (valid
at times=
tamp):<br>metadata_parse_version=3D1<br>metadata_feature_version=3D1<br>tim=
estamp=3D687358 (Sat Dec 27 08:42:04
2014)<br>host-id=3D2<br>score=3D2400<b=
r>maintenance=3DFalse<br>state=3DEngineDown<br><div><br></div><br>--=3D=3D
=
Host 3 status =3D=3D--<br><div><br></div>Status up-to-date
&n=
bsp; :
True<br>Hostname &nb=
sp;
&=
nbsp; : 10.0.0.92<br>Host ID
&nbs=
p; :
3<br>Engine status &nb=
sp;
: =
{"reason": "vm not running on<br>this host", "health":
"bad", "vm": "down",=
"detail": "unknown"}<br>Score
&n=
bsp;
: 0<br>Local ma=
intenance
: T=
rue<br>Host timestamp
&nbs=
p; : 681827<br>Extra metadata (valid at
timestamp):<br>metada=
ta_parse_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D681827 =
(Sat Dec 27 08:42:40
2014)<br>host-id=3D3<br>score=3D0<br>maintenance=3DTru=
e<br>state=3DLocalMaintenance<br>[root@compute2-1 vdsm]# tail -f /var/log/o=
virt-hosted-engine-ha/agent.log<br>MainThread::INFO::2014-12-27<br>08:42:41=
,109::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted=
Engine::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 240=
0)<br>MainThread::INFO::2014-12-27<br>08:42:51,198::state_decorators::124::=
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>Local m=
aintenance detected<br>MainThread::INFO::2014-12-27<br>08:42:51,420::hosted=
_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(sta=
rt_monitoring)<br>Current state LocalMaintenance (score: 0)<br>MainThread::=
INFO::2014-12-27<br>08:42:51,420::hosted_engine::332::ovirt_hosted_engine_h=
a.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host =
10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-27<br>08:43:01,=
507::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.Host=
edEngine::(check)<br>Local maintenance detected<br>MainThread::INFO::2014-1=
2-27<br>08:43:01,773::hosted_engine::327::ovirt_hosted_engine_ha.agent.host=
ed_engine.HostedEngine::(start_monitoring)<br>Current state LocalMaintenanc=
e (score: 0)<br>MainThread::INFO::2014-12-27<br>08:43:01,773::hosted_engine=
::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni=
toring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::I=
NFO::2014-12-27<br>08:43:11,859::state_decorators::124::ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine::(check)<br>Local maintenance detected=
<br>MainThread::INFO::2014-12-27<br>08:43:12,072::hosted_engine::327::ovirt=
_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>C=
urrent state LocalMaintenance (score: 0)<br>MainThread::INFO::2014-12-27<br=
08:43:12,072::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engi=
ne.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, s=
core:
2400)<br><div><br></div><br><div><br></div>[root@compute2-3
~]# tail =
-f /var/log/ovirt-hosted-engine-ha/agent.log<br>MainThread::INFO::2014-12-2=
7<br>11:36:28,855::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_=
engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: =
2, score: 2400)<br>MainThread::INFO::2014-12-27<br>11:36:39,130::hosted_eng=
ine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_m=
onitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::201=
4-12-27<br>11:36:39,130::hosted_engine::332::ovirt_hosted_engine_ha.agent.h=
osted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93=
(id: 2, score: 2400)<br>MainThread::INFO::2014-12-27<br>11:36:49,449::host=
ed_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(s=
tart_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INF=
O::2014-12-27<br>11:36:49,449::hosted_engine::332::ovirt_hosted_engine_ha.a=
gent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.=
0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-27<br>11:36:59,739=
::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngi=
ne::(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThrea=
d::INFO::2014-12-27<br>11:36:59,739::hosted_engine::332::ovirt_hosted_engin=
e_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote ho=
st 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-27<br>11:37:=
09,779::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin=
e::(consume)<br>Engine vm running on localhost<br>MainThread::INFO::2014-12=
-27<br>11:37:10,026::hosted_engine::327::ovirt_hosted_engine_ha.agent.hoste=
d_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (score:=
2400)<br>MainThread::INFO::2014-12-27<br>11:37:10,026::hosted_engine::332:=
:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring=
)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2=
014-12-27<br>11:37:20,331::hosted_engine::327::ovirt_hosted_engine_ha.agent=
.hosted_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (=
score: 2400)<br>MainThread::INFO::2014-12-27<br>11:37:20,331::hosted_engine=
::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni=
toring)<br>Best remote host 10.0.0.93 (id: 2, score:
2400)<br><div><br></di=
v><br>[root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.lo=
g<br>MainThread::INFO::2014-12-27<br>08:36:12,462::hosted_engine::332::ovir=
t_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>=
Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-1=
2-27<br>08:36:22,797::hosted_engine::327::ovirt_hosted_engine_ha.agent.host=
ed_engine.HostedEngine::(start_monitoring)<br>Current state EngineDown (sco=
re: 2400)<br>MainThread::INFO::2014-12-27<br>08:36:22,798::hosted_engine::3=
32::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitor=
ing)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO=
::2014-12-27<br>08:36:32,876::states::437::ovirt_hosted_engine_ha.agent.hos=
ted_engine.HostedEngine::(consume)<br>Engine vm is running on host 10.0.0.9=
4 (id 1)<br>MainThread::INFO::2014-12-27<br>08:36:33,169::hosted_engine::32=
7::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitori=
ng)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::2014-12-=
27<br>08:36:33,169::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted=
_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.94 (id:=
1, score: 2400)<br>MainThread::INFO::2014-12-27<br>08:36:43,567::hosted_en=
gine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_=
monitoring)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::=
2014-12-27<br>08:36:43,567::hosted_engine::332::ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0=
.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-27<br>08:36:53,858::h=
osted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:=
:(start_monitoring)<br>Current state EngineDown (score: 2400)<br>MainThread=
::INFO::2014-12-27<br>08:36:53,858::hosted_engine::332::ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote hos=
t 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-27<br>08:37:0=
4,028::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.Hoste=
dEngine::(refresh)<br>Global metadata: {'maintenance':
False}<br>MainThread=
::INFO::2014-12-27<br>08:37:04,028::state_machine::165::ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Host 10.0.0.94 (id 1): {=
'extra':<br>'metadata_parse_version=3D1\nmetadata_feature_version=3D1\ntime=
stamp=3D835987<br>(Sat Dec 27 11:37:30<br>2014)\nhost-id=3D1\nscore=3D2400\=
nmaintenance=3DFalse\nstate=3DEngineUp\n',<br>'hostname':
'10.0.0.94', 'ali=
ve': True, 'host-id': 1, 'engine-status':<br>{'health':
'good', 'vm': 'up',=
'detail': 'up'}, 'score': 2400,<br>'maintenance':
False, 'host-ts': 835987=
}<br>MainThread::INFO::2014-12-27<br>08:37:04,028::state_machine::165::ovir=
t_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Host 10.0=
.0.92 (id 3):
{'extra':<br>'metadata_parse_version=3D1\nmetadata_feature_ve=
rsion=3D1\ntimestamp=3D681528<br>(Sat Dec 27 08:37:41<br>2014)\nhost-id=3D3=
\nscore=3D0\nmaintenance=3DTrue\nstate=3DLocalMaintenance\n',<br>'hostname'=
: '10.0.0.92', 'alive': True, 'host-id': 3,
'engine-status':<br>{'reason': =
'vm not running on this host', 'health': 'bad',
'vm':<br>'down', 'detail': =
'unknown'}, 'score': 0, 'maintenance':
True,<br>'host-ts': 681528}<br>MainT=
hread::INFO::2014-12-27<br>08:37:04,028::state_machine::168::ovirt_hosted_e=
ngine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Local (id 2): {'eng=
ine-health': {'reason': 'vm not running on this<br>host',
'health': 'bad', =
'vm': 'down', 'detail': 'unknown'},
'bridge':<br>True, 'mem-free': 15300.0,=
'maintenance': False, 'cpu-load': 0.0215,<br>'gateway':
True}<br>MainThrea=
d::INFO::2014-12-27<br>08:37:04,265::hosted_engine::327::ovirt_hosted_engin=
e_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state =
EngineDown (score: 2400)<br>MainThread::INFO::2014-12-27<br>08:37:04,265::h=
osted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:=
:(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score:
2400)<br><=
div><br></div>Thanks,<br>Cong<br><div><br></div>On
2014/12/22, at 5:29, "Si=
mone Tiraboschi"
<stirabos@redhat.com<mailto:stirabos@redhat.com>&=
lt;mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>>
wro=
te:<br><div><br></div><br><div><br></div>-----
Original Message -----<br>Fr=
om: "Cong Yue"
<Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtele=
sis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@all=
iedtelesis.com>><br>To: "Simone Tiraboschi"
&lt;stirabos(a)redhat.com&l=
t;mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><mailt=
o:stirabos@redhat.com>><br>Cc:
users@ovirt.org<mailto:users@ovirt.=
org><mailto:users@ovirt.org><mailto:users@ovirt.org><br>Sent=
: Friday, December 19, 2014 7:22:10 PM<br>Subject: RE: [ovirt-users] VM fai=
lover with ovirt3.5<br><div><br></div>Thanks for the information.
This is t=
he log for my three ovirt nodes.<br>From the output of hosted-engine --vm-s=
tatus, it shows the engine state for<br>my 2nd and 3rd ovirt node is DOWN.<=
br>Is this the reason why VM failover not work in my
environment?<br><div><=
br></div>No, they looks ok: you can run the engine VM on single host at a t=
ime.<br><div><br></div>How can I make<br>also engine works
for my 2nd and 3=
rd ovit nodes?<br><div><br></div>If you put the host 1 in local
maintenance=
mode ( hosted-engine --set-maintenance --mode=3Dlocal ) the VM should migr=
ate to host 2; if you reactivate host 1 ( hosted-engine --set-maintenance -=
-mode=3Dnone ) and put host 2 in local maintenance mode the VM should migra=
te again.<br><div><br></div>Can you please try that and post the
logs if so=
mething is going
bad?<br><div><br></div><br>--<br>--=3D=3D Host 1
status =
=3D=3D--<br><div><br></div>Status up-to-date
&n=
bsp; : True<br>Hostname
&nb=
sp;
: 10.0.0=
.94<br>Host ID
&nbs=
p; : 1<br>Engine status
&nb=
sp;
: {"health": "go=
od", "vm": "up",<br>"detail":
"up"}<br>Score &n=
bsp;
:=
2400<br>Local maintenance
=
: False<br>Host timestamp
=
: 150475<br>Extra metadata
(valid at tim=
estamp):<br>metadata_parse_version=3D1<br>metadata_feature_version=3D1<br>t=
imestamp=3D150475 (Fri Dec 19 13:12:18 2014)<br>host-id=3D1<br>score=3D2400=
<br>maintenance=3DFalse<br>state=3DEngineUp<br><div><br></div><br>--=3D=3D
=
Host 2 status =3D=3D--<br><div><br></div>Status up-to-date
&n=
bsp; :
True<br>Hostname &nb=
sp;
&=
nbsp; : 10.0.0.93<br>Host ID
&nbs=
p; :
2<br>Engine status &nb=
sp;
: =
{"reason": "vm not running on<br>this host", "health":
"bad", "vm": "down",=
"detail": "unknown"}<br>Score
&n=
bsp;
: 2400<br>Local=
maintenance
=
: False<br>Host timestamp
=
: 1572<br>Extra metadata (valid at
timestamp):<br>meta=
data_parse_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D1572 =
(Fri Dec 19 10:12:18
2014)<br>host-id=3D2<br>score=3D2400<br>maintenance=3D=
False<br>state=3DEngineDown<br><div><br></div><br>--=3D=3D
Host 3 status =
=3D=3D--<br><div><br></div>Status up-to-date
&n=
bsp; : False<br>Hostname
&n=
bsp;
: 10.0.=
0.92<br>Host ID
&nb=
sp; : 3<br>Engine status
&n=
bsp;
: unknown stale=
-data<br>Score
&nbs=
p; : 2400<br>Local
maintenance &nb=
sp;
: False<br>Host =
timestamp
&n=
bsp; : 987<br>Extra metadata (valid at timestamp):<br>metadata_parse_versio=
n=3D1<br>metadata_feature_version=3D1<br>timestamp=3D987 (Fri Dec 19 10:09:=
58
2014)<br>host-id=3D3<br>score=3D2400<br>maintenance=3DFalse<br>state=3DE=
ngineDown<br><div><br></div>--<br>And the
/var/log/ovirt-hosted-engine-ha/a=
gent.log for three ovirt nodes are<br>as
follows:<br>--<br>10.0.0.94(hosted=
-engine-1)<br>---<br>MainThread::INFO::2014-12-19<br>13:09:33,716::hosted_e=
ngine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start=
_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::2=
014-12-19<br>13:09:33,716::hosted_engine::332::ovirt_hosted_engine_ha.agent=
.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.=
93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:09:44,017::ho=
sted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::=
(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::I=
NFO::2014-12-19<br>13:09:44,017::hosted_engine::332::ovirt_hosted_engine_ha=
.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 1=
0.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:09:54,3=
03::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEn=
gine::(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThr=
ead::INFO::2014-12-19<br>13:09:54,303::hosted_engine::332::ovirt_hosted_eng=
ine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote =
host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:1=
0:04,342::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEng=
ine::(consume)<br>Engine vm running on localhost<br>MainThread::INFO::2014-=
12-19<br>13:10:04,617::hosted_engine::327::ovirt_hosted_engine_ha.agent.hos=
ted_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (scor=
e: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:04,617::hosted_engine::33=
2::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitori=
ng)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO:=
:2014-12-19<br>13:10:14,657::state_machine::160::ovirt_hosted_engine_ha.age=
nt.hosted_engine.HostedEngine::(refresh)<br>Global metadata:
{'maintenance'=
: False}<br>MainThread::INFO::2014-12-19<br>13:10:14,657::state_machine::16=
5::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Ho=
st 10.0.0.93 (id 2):
{'extra':<br>'metadata_parse_version=3D1\nmetadata_fea=
ture_version=3D1\ntimestamp=3D1448<br>(Fri Dec 19 10:10:14<br>2014)\nhost-i=
d=3D2\nscore=3D2400\nmaintenance=3DFalse\nstate=3DEngineDown\n',<br>'hostna=
me': '10.0.0.93', 'alive': True, 'host-id': 2,
'engine-status':<br>{'reason=
': 'vm not running on this host', 'health': 'bad',
'vm':<br>'down', 'detail=
': 'unknown'}, 'score': 2400, 'maintenance':
False,<br>'host-ts': 1448}<br>=
MainThread::INFO::2014-12-19<br>13:10:14,657::state_machine::165::ovirt_hos=
ted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Host 10.0.0.92=
(id 3):
{'extra':<br>'metadata_parse_version=3D1\nmetadata_feature_version=
=3D1\ntimestamp=3D987<br>(Fri Dec 19 10:09:58<br>2014)\nhost-id=3D3\nscore=
=3D2400\nmaintenance=3DFalse\nstate=3DEngineDown\n',<br>'hostname':
'10.0.0=
.92', 'alive': True, 'host-id': 3,
'engine-status':<br>{'reason': 'vm not r=
unning on this host', 'health': 'bad',
'vm':<br>'down', 'detail': 'unknown'=
}, 'score': 2400, 'maintenance': False,<br>'host-ts':
987}<br>MainThread::I=
NFO::2014-12-19<br>13:10:14,658::state_machine::168::ovirt_hosted_engine_ha=
.agent.hosted_engine.HostedEngine::(refresh)<br>Local (id 1): {'engine-heal=
th': {'health': 'good', 'vm':
'up',<br>'detail': 'up'}, 'bridge': True, 'me=
m-free': 1079.0, 'maintenance':<br>False, 'cpu-load': 0.0269,
'gateway': Tr=
ue}<br>MainThread::INFO::2014-12-19<br>13:10:14,904::hosted_engine::327::ov=
irt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<b=
r>Current state EngineUp (score:
2400)<br>MainThread::INFO::2014-12-19<br>1=
3:10:14,904::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine=
.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2, sco=
re: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:25,210::hosted_engine::3=
27::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitor=
ing)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::2014-12-1=
9<br>13:10:25,210::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_=
engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: =
2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:35,499::hosted_eng=
ine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_m=
onitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::201=
4-12-19<br>13:10:35,499::hosted_engine::332::ovirt_hosted_engine_ha.agent.h=
osted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93=
(id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:45,784::host=
ed_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(s=
tart_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INF=
O::2014-12-19<br>13:10:45,785::hosted_engine::332::ovirt_hosted_engine_ha.a=
gent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.=
0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:56,070=
::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngi=
ne::(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThrea=
d::INFO::2014-12-19<br>13:10:56,070::hosted_engine::332::ovirt_hosted_engin=
e_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote ho=
st 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:11:=
06,109::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin=
e::(consume)<br>Engine vm running on localhost<br>MainThread::INFO::2014-12=
-19<br>13:11:06,359::hosted_engine::327::ovirt_hosted_engine_ha.agent.hoste=
d_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (score:=
2400)<br>MainThread::INFO::2014-12-19<br>13:11:06,359::hosted_engine::332:=
:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring=
)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2=
014-12-19<br>13:11:16,658::hosted_engine::327::ovirt_hosted_engine_ha.agent=
.hosted_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (=
score: 2400)<br>MainThread::INFO::2014-12-19<br>13:11:16,658::hosted_engine=
::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni=
toring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::I=
NFO::2014-12-19<br>13:11:26,991::hosted_engine::327::ovirt_hosted_engine_ha=
.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state Engi=
neUp (score: 2400)<br>MainThread::INFO::2014-12-19<br>13:11:26,991::hosted_=
engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(star=
t_monitoring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThr=
ead::INFO::2014-12-19<br>13:11:37,341::hosted_engine::327::ovirt_hosted_eng=
ine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current stat=
e EngineUp (score: 2400)<br>MainThread::INFO::2014-12-19<br>13:11:37,341::h=
osted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:=
:(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>-=
---<br><div><br></div>10.0.0.93
(hosted-engine-2)<br>MainThread::INFO::2014=
-12-19<br>10:12:18,339::hosted_engine::327::ovirt_hosted_engine_ha.agent.ho=
sted_engine.HostedEngine::(start_monitoring)<br>Current state EngineDown (s=
core: 2400)<br>MainThread::INFO::2014-12-19<br>10:12:18,339::hosted_engine:=
:332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monit=
oring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::IN=
FO::2014-12-19<br>10:12:28,651::hosted_engine::327::ovirt_hosted_engine_ha.=
agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state Engin=
eDown (score: 2400)<br>MainThread::INFO::2014-12-19<br>10:12:28,652::hosted=
_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(sta=
rt_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainTh=
read::INFO::2014-12-19<br>10:12:39,010::hosted_engine::327::ovirt_hosted_en=
gine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current sta=
te EngineDown (score: 2400)<br>MainThread::INFO::2014-12-19<br>10:12:39,010=
::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngi=
ne::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<b=
r>MainThread::INFO::2014-12-19<br>10:12:49,338::hosted_engine::327::ovirt_h=
osted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Cur=
rent state EngineDown (score: 2400)<br>MainThread::INFO::2014-12-19<br>10:1=
2:49,338::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Ho=
stedEngine::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score:=
2400)<br>MainThread::INFO::2014-12-19<br>10:12:59,642::hosted_engine::327:=
:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring=
)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::2014-12-19=
<br>10:12:59,642::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_e=
ngine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1=
, score: 2400)<br>MainThread::INFO::2014-12-19<br>10:13:10,010::hosted_engi=
ne::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_mo=
nitoring)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::20=
14-12-19<br>10:13:10,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.=
hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.9=
4 (id: 1, score:
2400)<br><div><br></div><br>10.0.0.92(hosted-engine-3)<br>=
same as 10.0.0.93<br>--<br><div><br></div>-----Original
Message-----<br>Fro=
m: Simone Tiraboschi [mailto:stirabos@redhat.com]<br>Sent: Friday, December=
19, 2014 12:28 AM<br>To: Yue, Cong<br>Cc:
users@ovirt.org<mailto:users@=
ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org><b=
r>Subject: Re: [ovirt-users] VM failover with
ovirt3.5<br><div><br></div><b=
r><div><br></div>----- Original Message -----<br>From:
"Cong Yue" <Cong_=
Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Co=
ng_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>><b=
r>To:
users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.o=
rg><mailto:users@ovirt.org><br>Sent: Friday, December 19, 2014
2:1=
4:33 AM<br>Subject: [ovirt-users] VM failover with
ovirt3.5<br><div><br></d=
iv><br><div><br></div>Hi<br><div><br></div><br><div><br></div>In
my environ=
ment, I have 3 ovirt nodes as one cluster. And on top of<br>host-1, there i=
s one vm to host ovirt engine.<br><div><br></div>Also I have one
external s=
torage for the cluster to use as data domain<br>of engine and
data.<br><div=
<br></div>I confirmed live migration works well in my
environment.<br><div=
<br></div>But it seems very buggy for VM failover if I try to force to shu=
t down<br>one ovirt node. Sometimes the VM in the node which is shutdown ca=
n<br>migrate to other host, but it take more than several
minutes.<br><div>=
<br></div>Sometimes, it can not migrate at all. Sometimes, only when the ho=
st is<br>back, the VM is beginning to
move.<br><div><br></div>Can you pleas=
e check or share the logs under
/var/log/ovirt-hosted-engine-ha/<br>?<br><d=
iv><br></div>Is there some documentation to explain how VM failover is
work=
ing? And<br>is there some bugs reported related with
this?<br><div><br></di=
v>http://www.ovirt.org/Features/Self_Hosted_Engine#Agent_State_Diagram...
div><br></div>Thanks in
advance,<br><div><br></div>Cong<br><div><br></div><=
br><div><br></div><br>This e-mail message is for the sole use
of the intend=
ed recipient(s)<br>and may contain confidential and privileged information.=
Any<br>unauthorized review, use, disclosure or distribution is prohibited.=
If<br>you are not the intended recipient, please contact the sender by rep=
ly<br>e-mail and destroy all copies of the original message. If you are the=
<br>intended recipient, please be advised that the content of this message<=
br>is subject to access, review and disclosure by the sender's e-mail Syste=
m<br>Administrator.<br><div><br></div>_____________________________________=
__________<br>Users mailing
list<br>Users@ovirt.org<mailto:Users@ovirt.o=
rg><mailto:Users@ovirt.org><mailto:Users@ovirt.org><br>http:=
//lists.ovirt.org/mailman/listinfo/users<br><div><br></div>This
e-mail mess=
age is for the sole use of the intended recipient(s) and may<br>contain con=
fidential and privileged information. Any unauthorized review,<br>use, disc=
losure or distribution is prohibited. If you are not the intended<br>recipi=
ent, please contact the sender by reply e-mail and destroy all copies<br>of=
the original message. If you are the intended recipient, please be<br>advi=
sed that the content of this message is subject to access, review and<br>di=
sclosure by the sender's e-mail System
Administrator.<br><div><br></div><br=
This e-mail message is for the sole use of the intended recipient(s)
and m=
ay contain confidential and privileged information. Any unauthorized review=
, use, disclosure or distribution is prohibited. If you are not the intende=
d recipient, please contact the sender by reply e-mail and destroy all copi=
es of the original message. If you are the intended recipient, please be ad=
vised that the content of this message is subject to access, review and dis=
closure by the sender's e-mail System Administrator.<br>___________________=
____________________________<br>Users mailing
list<br>Users(a)ovirt.org&lt;ma=
ilto:Users@ovirt.org><mailto:Users@ovirt.org><mailto:Users@ovir=
t.org><br>http://lists.ovirt.org/mailman/listinfo/users<b...
v>________________________________<br>This e-mail message is for the sole u=
se of the intended recipient(s) and may contain confidential and privileged=
information. Any unauthorized review, use, disclosure or distribution is p=
rohibited. If you are not the intended recipient, please contact the sender=
by reply e-mail and destroy all copies of the original message. If you are=
the intended recipient, please be advised that the content of this message=
is subject to access, review and disclosure by the sender's e-mail System =
Administrator.<br><div><br></div>________________________________<br>This
e=
-mail message is for the sole use of the intended recipient(s) and may cont=
ain confidential and privileged information. Any unauthorized review, use, =
disclosure or distribution is prohibited. If you are not the intended recip=
ient, please contact the sender by reply e-mail and destroy all copies of t=
he original message. If you are the intended recipient, please be advised t=
hat the content of this message is subject to access, review and disclosure=
by the sender's e-mail System
Administrator.<br><div><br></div>___________=
_____________________<br>This e-mail message is for the sole use of the int=
ended recipient(s) and may contain confidential and privileged information.=
Any unauthorized review, use, disclosure or distribution is prohibited. If=
you are not the intended recipient, please contact the sender by reply e-m=
ail and destroy all copies of the original message. If you are the intended=
recipient, please be advised that the content of this message is subject t=
o access, review and disclosure by the sender's e-mail System Administrator=
.<br>-------------- next part --------------<br>An HTML attachment was scru=
bbed...<br>URL: <http://lists.ovirt.org/pipermail/users/attachments/2014=
1229/4ec6cc13/attachment.html><br><div><br></div>-----------------------=
-------<br><div><br></div>_______________________________________________<b=
r>Users mailing
list<br>Users@ovirt.org<br>http://lists.ovirt.org/mailman/l=
istinfo/users<br><div><br></div><br>End of Users Digest, Vol
39, Issue 171<=
br>**************************************<br></div><div><br></div></div></b=
ody></html>
------=_Part_1882534_1428653136.1419879250032--