Re: [ovirt-users] VM failover with ovirt3.5

29 Dec 2014

      ------=_Part_1875460_365779577.1419876418683
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Hi, 
Your guest-vm have to be defined as " Highly Available" 
Highly Available 

Select this check box if the virtual machine is to be highly available. For example, in cases of host maintenance or failure, the virtual machine is automatically moved to or re-launched on another host. If the host is manually shut down by the system administrator, the virtual machine is not automatically moved to another host. 
Note that this option is unavailable if the Migration Options setting in the Hosts tab is set to either Allow manual migration only or No migration . For a virtual machine to be highly available, it must be possible for the Manager to migrate the virtual machine to other available hosts as necessary. 

Thanks in advance. 

Best regards, 
Nikolai 
____________________ 
Nikolai Sednev 
Senior Quality Engineer at Compute team 
Red Hat Israel 
34 Jerusalem Road, 
Ra'anana, Israel 43501 

Tel: +972 9 7692043 
Mobile: +972 52 7342734 
Email: nsednev@redhat.com 
IRC: nsednev 

----- Original Message -----

From: users-request@ovirt.org 
To: users@ovirt.org 
Sent: Monday, December 29, 2014 7:50:07 PM 
Subject: Users Digest, Vol 39, Issue 169 

Send Users mailing list submissions to 
users@ovirt.org 

To subscribe or unsubscribe via the World Wide Web, visit 
http://lists.ovirt.org/mailman/listinfo/users 
or, via email, send a message with subject or body 'help' to 
users-request@ovirt.org 

You can reach the person managing the list at 
users-owner@ovirt.org 

When replying, please edit your Subject line so it is more specific 
than "Re: Contents of Users digest..." 

Today's Topics: 

1. Re: VM failover with ovirt3.5 (Yue, Cong) 

---------------------------------------------------------------------- 

Message: 1 
Date: Mon, 29 Dec 2014 09:49:58 -0800 
From: "Yue, Cong" <Cong_Yue@alliedtelesis.com> 
To: Artyom Lukianov <alukiano@redhat.com> 
Cc: "users@ovirt.org" <users@ovirt.org> 
Subject: Re: [ovirt-users] VM failover with ovirt3.5 
Message-ID: <11A51118-8B03-41FE-8FD0-C81AC8897EF6@alliedtelesis.com> 
Content-Type: text/plain; charset="us-ascii" 

Thanks for detailed explanation. Do you mean only HE VM can be failover? I want to have a try with the VM on any host to check whether VM can be failover to other host automatically like VMware or Xenserver? 
I will have a try as you advised and provide the log for your further advice. 

Thanks, 
Cong
...
On 2014/12/29, at 8:43, "Artyom Lukianov" <alukiano@redhat.com> wrote:
I see that HE vm run on host with ip 10.0.0.94, and two another hosts in "Local Maintenance" state, so vm will not migrate to any of them, can you try disable local maintenance on all hosts in HE environment and after enable "local maintenance" on host where HE vm run, and provide also output of hosted-engine --vm-status. 
Failover works in next way: 
1) if host where run HE vm have score less by 800 that some other host in HE environment, HE vm will migrate on host with best score 
2) if something happen to vm(kernel panic, crash of service...), agent will restart HE vm on another host in HE environment with positive score 
3) if put to local maintenance host with HE vm, vm will migrate to another host with positive score 
Thanks.
----- Original Message ----- 
From: "Cong Yue" <Cong_Yue@alliedtelesis.com> 
To: "Artyom Lukianov" <alukiano@redhat.com> 
Cc: "Simone Tiraboschi" <stirabos@redhat.com>, users@ovirt.org 
Sent: Monday, December 29, 2014 6:30:42 PM 
Subject: Re: [ovirt-users] VM failover with ovirt3.5
Thanks and the --vm-status log is as follows: 
[root@compute2-2 ~]# hosted-engine --vm-status
--== Host 1 status ==--
Status up-to-date : True 
Hostname : 10.0.0.94 
Host ID : 1 
Engine status : {"health": "good", "vm": "up", 
"detail": "up"} 
Score : 2400 
Local maintenance : False 
Host timestamp : 1008087 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=1008087<tel:1008087> (Mon Dec 29 11:25:51 2014) 
host-id=1 
score=2400 
maintenance=False 
state=EngineUp
--== Host 2 status ==--
Status up-to-date : True 
Hostname : 10.0.0.93 
Host ID : 2 
Engine status : {"reason": "vm not running on 
this host", "health": "bad", "vm": "down", "detail": "unknown"} 
Score : 0 
Local maintenance : True 
Host timestamp : 859142 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=859142 (Mon Dec 29 08:25:08 2014) 
host-id=2 
score=0 
maintenance=True 
state=LocalMaintenance
--== Host 3 status ==--
Status up-to-date : True 
Hostname : 10.0.0.92 
Host ID : 3 
Engine status : {"reason": "vm not running on 
this host", "health": "bad", "vm": "down", "detail": "unknown"} 
Score : 0 
Local maintenance : True 
Host timestamp : 853615 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=853615 (Mon Dec 29 08:25:57 2014) 
host-id=3 
score=0 
maintenance=True 
state=LocalMaintenance 
You have new mail in /var/spool/mail/root 
[root@compute2-2 ~]#
Could you please explain how VM failover works inside ovirt? Is there any other debug option I can enable to check the problem?
Thanks, 
Cong
On 2014/12/29, at 1:39, "Artyom Lukianov" <alukiano@redhat.com<mailto:alukiano@redhat.com>> wrote:
Can you also provide output of hosted-engine --vm-status please, previous time it was useful, because I do not see something unusual. 
Thanks
----- Original Message ----- 
From: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com>> 
To: "Artyom Lukianov" <alukiano@redhat.com<mailto:alukiano@redhat.com>> 
Cc: "Simone Tiraboschi" <stirabos@redhat.com<mailto:stirabos@redhat.com>>, users@ovirt.org<mailto:users@ovirt.org> 
Sent: Monday, December 29, 2014 7:15:24 AM 
Subject: Re: [ovirt-users] VM failover with ovirt3.5
Also I change the maintenance mode to local in another host. But also the VM in this host can not be migrated. The logs are as follows.
[root@compute2-2 ~]# hosted-engine --set-maintenance --mode=local 
[root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log 
MainThread::INFO::2014-12-28 
21:09:04,184::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-28 
21:09:14,603::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-28 
21:09:14,603::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-28 
21:09:24,903::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-28 
21:09:24,904::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-28 
21:09:35,026::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) 
Engine vm is running on host 10.0.0.94 (id 1) 
MainThread::INFO::2014-12-28 
21:09:35,236::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-28 
21:09:35,236::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-28 
21:09:45,604::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-28 
21:09:45,604::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-28 
21:09:55,691::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) 
Local maintenance detected 
MainThread::INFO::2014-12-28 
21:09:55,701::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) 
Trying: notify time=1419829795.7 type=state_transition 
detail=EngineDown-LocalMaintenance hostname='compute2-2' 
MainThread::INFO::2014-12-28 
21:09:55,761::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) 
Success, was notification of state_transition 
(EngineDown-LocalMaintenance) sent? sent 
MainThread::INFO::2014-12-28 
21:09:55,990::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) 
Score is 0 due to local maintenance mode 
MainThread::INFO::2014-12-28 
21:09:55,990::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state LocalMaintenance (score: 0) 
MainThread::INFO::2014-12-28 
21:09:55,991::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
^C 
You have new mail in /var/spool/mail/root 
[root@compute2-2 ~]# ps -ef | grep qemu 
root 18420 2777 0 21:10<x-apple-data-detectors://39> pts/0 00:00:00<x-apple-data-detectors://40> grep --color=auto qemu 
qemu 29809 1 0 Dec19 ? 01:17:20 /usr/libexec/qemu-kvm 
-name testvm2-2 -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Nehalem 
-m 500 -realtime mlock=off -smp 
1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 
c31e97d0-135e-42da-9954-162b5228dce3 -smbios 
type=1,manufacturer=oVirt,product=oVirt 
Node,version=7-0.1406.el7.centos.2.5,serial=4C4C4544-0059-3610-8033-B4C04F395931,uuid=c31e97d0-135e-42da-9954-162b5228dce3 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/testvm2-2.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=2014-12-19T20:17:17<x-apple-data-detectors://42>,driftfix=slew -no-kvm-pit-reinjection 
-no-hpet -no-shutdown -boot strict=on -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device 
virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= 
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 
-drive file=/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/5cbeb8c9-4f04-48d0-a5eb-78c49187c550/a0570e8c-9867-4ec4-818f-11e102fc4f9b,if=none,id=drive-virtio-disk0,format=qcow2,serial=5cbeb8c9-4f04-48d0-a5eb-78c49187c550,cache=none,werror=stop,rerror=stop,aio=threads 
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:db:94:00,bus=pci.0,addr=0x3 
-chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3.com.redhat.rhevm.vdsm,server,nowait 
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm 
-chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3.org.qemu.guest_agent.0,server,nowait 
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 
-chardev spicevmc,id=charchannel2,name=vdagent -device 
virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 
-spice tls-port=5901,addr=10.0.0.93,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on 
-k en-us -vga qxl -global qxl-vga.ram_size=67108864<tel:67108864> -global 
qxl-vga.vram_size=33554432<tel:33554432> -incoming tcp:[::]:49152 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 
[root@compute2-2 ~]#
Thanks, 
Cong
On 2014/12/28, at 20:53, "Yue, Cong" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> wrote:
I checked it again and confirmed there is one guest VM is running on the top of this host. The log is as follows:
[root@compute2-1 vdsm]# ps -ef | grep qemu 
qemu 2983 846 0 Dec19 ? 00:00:00<x-apple-data-detectors://0> [supervdsmServer] <defunct> 
root 5489 3053 0 20:49<x-apple-data-detectors://1> pts/0 00:00:00<x-apple-data-detectors://2> grep --color=auto qemu 
qemu 26128 1 0 Dec19 ? 01:09:19 /usr/libexec/qemu-kvm 
-name testvm2 -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Nehalem -m 
500 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 
-uuid e46bca87-4df5-4287-844b-90a26fccef33 -smbios 
type=1,manufacturer=oVirt,product=oVirt 
Node,version=7-0.1406.el7.centos.2.5,serial=4C4C4544-0030-3310-8059-B8C04F585231,uuid=e46bca87-4df5-4287-844b-90a26fccef33 
-no-user-config -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/testvm2.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc 
base=2014-12-19T20:18:01<x-apple-data-detectors://4>,driftfix=slew -no-kvm-pit-reinjection 
-no-hpet -no-shutdown -boot strict=on -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device 
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device 
virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 
-drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= 
-device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 
-drive file=/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/b4b5426b-95e3-41af-b286-da245891cdaf/0f688d49-97e3-4f1d-84d4-ac1432d903b3,if=none,id=drive-virtio-disk0,format=qcow2,serial=b4b5426b-95e3-41af-b286-da245891cdaf,cache=none,werror=stop,rerror=stop,aio=threads 
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:db:94:01,bus=pci.0,addr=0x3 
-chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef33.com.redhat.rhevm.vdsm,server,nowait 
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm 
-chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef33.org.qemu.guest_agent.0,server,nowait 
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 
-chardev spicevmc,id=charchannel2,name=vdagent -device 
virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 
-spice tls-port=5900,addr=10.0.0.92,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on 
-k en-us -vga qxl -global qxl-vga.ram_size=67108864<tel:67108864> -global 
qxl-vga.vram_size=33554432<tel:33554432> -incoming tcp:[::]:49152 -device 
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 
[root@compute2-1 vdsm]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log 
MainThread::INFO::2014-12-28 
20:49:27,315::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) 
Local maintenance detected 
MainThread::INFO::2014-12-28 
20:49:27,646::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state LocalMaintenance (score: 0) 
MainThread::INFO::2014-12-28 
20:49:27,646::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-28 
20:49:37,732::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) 
Local maintenance detected 
MainThread::INFO::2014-12-28 
20:49:37,961::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state LocalMaintenance (score: 0) 
MainThread::INFO::2014-12-28 
20:49:37,961::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-28 
20:49:48,048::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) 
Local maintenance detected 
MainThread::INFO::2014-12-28 
20:49:48,319::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) 
Score is 0 due to local maintenance mode 
MainThread::INFO::2014-12-28 
20:49:48,319::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state LocalMaintenance (score: 0) 
MainThread::INFO::2014-12-28 
20:49:48,319::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400)
Thanks, 
Cong
On 2014/12/28, at 3:46, "Artyom Lukianov" <alukiano@redhat.com<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>> wrote:
I see that you set local maintenance on host3 that do not have engine vm on it, so it nothing to migrate from this host. 
If you set local maintenance on host1, vm must migrate to another host with positive score. 
Thanks
----- Original Message ----- 
From: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> 
To: "Simone Tiraboschi" <stirabos@redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>> 
Cc: users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org> 
Sent: Saturday, December 27, 2014 6:58:32 PM 
Subject: Re: [ovirt-users] VM failover with ovirt3.5
Hi
I had a try with "hosted-engine --set-maintence --mode=local" on 
compute2-1, which is host 3 in my cluster. From the log, it shows 
maintence mode is dectected, but migration does not happen.
The logs are as follows. Is there any other config I need to check?
[root@compute2-1 vdsm]# hosted-engine --vm-status
--== Host 1 status ==-
Status up-to-date : True 
Hostname : 10.0.0.94 
Host ID : 1 
Engine status : {"health": "good", "vm": "up", 
"detail": "up"} 
Score : 2400 
Local maintenance : False 
Host timestamp : 836296 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=836296 (Sat Dec 27 11:42:39 2014) 
host-id=1 
score=2400 
maintenance=False 
state=EngineUp
--== Host 2 status ==--
Status up-to-date : True 
Hostname : 10.0.0.93 
Host ID : 2 
Engine status : {"reason": "vm not running on 
this host", "health": "bad", "vm": "down", "detail": "unknown"} 
Score : 2400 
Local maintenance : False 
Host timestamp : 687358 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=687358 (Sat Dec 27 08:42:04 2014) 
host-id=2 
score=2400 
maintenance=False 
state=EngineDown
--== Host 3 status ==--
Status up-to-date : True 
Hostname : 10.0.0.92 
Host ID : 3 
Engine status : {"reason": "vm not running on 
this host", "health": "bad", "vm": "down", "detail": "unknown"} 
Score : 0 
Local maintenance : True 
Host timestamp : 681827 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=681827 (Sat Dec 27 08:42:40 2014) 
host-id=3 
score=0 
maintenance=True 
state=LocalMaintenance 
[root@compute2-1 vdsm]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log 
MainThread::INFO::2014-12-27 
08:42:41,109::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-27 
08:42:51,198::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) 
Local maintenance detected 
MainThread::INFO::2014-12-27 
08:42:51,420::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state LocalMaintenance (score: 0) 
MainThread::INFO::2014-12-27 
08:42:51,420::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-27 
08:43:01,507::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) 
Local maintenance detected 
MainThread::INFO::2014-12-27 
08:43:01,773::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state LocalMaintenance (score: 0) 
MainThread::INFO::2014-12-27 
08:43:01,773::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-27 
08:43:11,859::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) 
Local maintenance detected 
MainThread::INFO::2014-12-27 
08:43:12,072::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state LocalMaintenance (score: 0) 
MainThread::INFO::2014-12-27 
08:43:12,072::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400)
[root@compute2-3 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log 
MainThread::INFO::2014-12-27 
11:36:28,855::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-27 
11:36:39,130::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-27 
11:36:39,130::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-27 
11:36:49,449::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-27 
11:36:49,449::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-27 
11:36:59,739::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-27 
11:36:59,739::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-27 
11:37:09,779::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) 
Engine vm running on localhost 
MainThread::INFO::2014-12-27 
11:37:10,026::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-27 
11:37:10,026::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-27 
11:37:20,331::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-27 
11:37:20,331::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400)
[root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log 
MainThread::INFO::2014-12-27 
08:36:12,462::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-27 
08:36:22,797::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-27 
08:36:22,798::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-27 
08:36:32,876::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) 
Engine vm is running on host 10.0.0.94 (id 1) 
MainThread::INFO::2014-12-27 
08:36:33,169::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-27 
08:36:33,169::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-27 
08:36:43,567::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-27 
08:36:43,567::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-27 
08:36:53,858::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-27 
08:36:53,858::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-27 
08:37:04,028::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) 
Global metadata: {'maintenance': False} 
MainThread::INFO::2014-12-27 
08:37:04,028::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) 
Host 10.0.0.94 (id 1): {'extra': 
'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=835987 
(Sat Dec 27 11:37:30 
2014)\nhost-id=1\nscore=2400\nmaintenance=False\nstate=EngineUp\n', 
'hostname': '10.0.0.94', 'alive': True, 'host-id': 1, 'engine-status': 
{'health': 'good', 'vm': 'up', 'detail': 'up'}, 'score': 2400, 
'maintenance': False, 'host-ts': 835987} 
MainThread::INFO::2014-12-27 
08:37:04,028::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) 
Host 10.0.0.92 (id 3): {'extra': 
'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=681528 
(Sat Dec 27 08:37:41 
2014)\nhost-id=3\nscore=0\nmaintenance=True\nstate=LocalMaintenance\n', 
'hostname': '10.0.0.92', 'alive': True, 'host-id': 3, 'engine-status': 
{'reason': 'vm not running on this host', 'health': 'bad', 'vm': 
'down', 'detail': 'unknown'}, 'score': 0, 'maintenance': True, 
'host-ts': 681528} 
MainThread::INFO::2014-12-27 
08:37:04,028::state_machine::168::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) 
Local (id 2): {'engine-health': {'reason': 'vm not running on this 
host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': 
True, 'mem-free': 15300.0, 'maintenance': False, 'cpu-load': 0.0215, 
'gateway': True} 
MainThread::INFO::2014-12-27 
08:37:04,265::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-27 
08:37:04,265::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400)
Thanks, 
Cong
On 2014/12/22, at 5:29, "Simone Tiraboschi" <stirabos@redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>> wrote:
----- Original Message ----- 
From: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> 
To: "Simone Tiraboschi" <stirabos@redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>> 
Cc: users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org> 
Sent: Friday, December 19, 2014 7:22:10 PM 
Subject: RE: [ovirt-users] VM failover with ovirt3.5
Thanks for the information. This is the log for my three ovirt nodes. 
From the output of hosted-engine --vm-status, it shows the engine state for 
my 2nd and 3rd ovirt node is DOWN. 
Is this the reason why VM failover not work in my environment?
No, they looks ok: you can run the engine VM on single host at a time.
How can I make 
also engine works for my 2nd and 3rd ovit nodes?
If you put the host 1 in local maintenance mode ( hosted-engine --set-maintenance --mode=local ) the VM should migrate to host 2; if you reactivate host 1 ( hosted-engine --set-maintenance --mode=none ) and put host 2 in local maintenance mode the VM should migrate again.
Can you please try that and post the logs if something is going bad?
-- 
--== Host 1 status ==--
Status up-to-date : True 
Hostname : 10.0.0.94 
Host ID : 1 
Engine status : {"health": "good", "vm": "up", 
"detail": "up"} 
Score : 2400 
Local maintenance : False 
Host timestamp : 150475 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=150475 (Fri Dec 19 13:12:18 2014) 
host-id=1 
score=2400 
maintenance=False 
state=EngineUp
--== Host 2 status ==--
Status up-to-date : True 
Hostname : 10.0.0.93 
Host ID : 2 
Engine status : {"reason": "vm not running on 
this host", "health": "bad", "vm": "down", "detail": "unknown"} 
Score : 2400 
Local maintenance : False 
Host timestamp : 1572 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=1572 (Fri Dec 19 10:12:18 2014) 
host-id=2 
score=2400 
maintenance=False 
state=EngineDown
--== Host 3 status ==--
Status up-to-date : False 
Hostname : 10.0.0.92 
Host ID : 3 
Engine status : unknown stale-data 
Score : 2400 
Local maintenance : False 
Host timestamp : 987 
Extra metadata (valid at timestamp): 
metadata_parse_version=1 
metadata_feature_version=1 
timestamp=987 (Fri Dec 19 10:09:58 2014) 
host-id=3 
score=2400 
maintenance=False 
state=EngineDown
-- 
And the /var/log/ovirt-hosted-engine-ha/agent.log for three ovirt nodes are 
as follows: 
-- 
10.0.0.94(hosted-engine-1) 
--- 
MainThread::INFO::2014-12-19 
13:09:33,716::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:09:33,716::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:09:44,017::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:09:44,017::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:09:54,303::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:09:54,303::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:10:04,342::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) 
Engine vm running on localhost 
MainThread::INFO::2014-12-19 
13:10:04,617::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:10:04,617::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:10:14,657::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) 
Global metadata: {'maintenance': False} 
MainThread::INFO::2014-12-19 
13:10:14,657::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) 
Host 10.0.0.93 (id 2): {'extra': 
'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1448 
(Fri Dec 19 10:10:14 
2014)\nhost-id=2\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 
'hostname': '10.0.0.93', 'alive': True, 'host-id': 2, 'engine-status': 
{'reason': 'vm not running on this host', 'health': 'bad', 'vm': 
'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 
'host-ts': 1448} 
MainThread::INFO::2014-12-19 
13:10:14,657::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) 
Host 10.0.0.92 (id 3): {'extra': 
'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=987 
(Fri Dec 19 10:09:58 
2014)\nhost-id=3\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 
'hostname': '10.0.0.92', 'alive': True, 'host-id': 3, 'engine-status': 
{'reason': 'vm not running on this host', 'health': 'bad', 'vm': 
'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 
'host-ts': 987} 
MainThread::INFO::2014-12-19 
13:10:14,658::state_machine::168::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) 
Local (id 1): {'engine-health': {'health': 'good', 'vm': 'up', 
'detail': 'up'}, 'bridge': True, 'mem-free': 1079.0, 'maintenance': 
False, 'cpu-load': 0.0269, 'gateway': True} 
MainThread::INFO::2014-12-19 
13:10:14,904::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:10:14,904::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:10:25,210::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:10:25,210::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:10:35,499::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:10:35,499::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:10:45,784::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:10:45,785::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:10:56,070::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:10:56,070::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:11:06,109::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) 
Engine vm running on localhost 
MainThread::INFO::2014-12-19 
13:11:06,359::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:11:06,359::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:11:16,658::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:11:16,658::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:11:26,991::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:11:26,991::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
MainThread::INFO::2014-12-19 
13:11:37,341::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineUp (score: 2400) 
MainThread::INFO::2014-12-19 
13:11:37,341::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.93 (id: 2, score: 2400) 
----
10.0.0.93 (hosted-engine-2) 
MainThread::INFO::2014-12-19 
10:12:18,339::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-19 
10:12:18,339::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-19 
10:12:28,651::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-19 
10:12:28,652::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-19 
10:12:39,010::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-19 
10:12:39,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-19 
10:12:49,338::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-19 
10:12:49,338::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-19 
10:12:59,642::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-19 
10:12:59,642::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400) 
MainThread::INFO::2014-12-19 
10:13:10,010::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Current state EngineDown (score: 2400) 
MainThread::INFO::2014-12-19 
10:13:10,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) 
Best remote host 10.0.0.94 (id: 1, score: 2400)
10.0.0.92(hosted-engine-3) 
same as 10.0.0.93 
--
-----Original Message----- 
From: Simone Tiraboschi [mailto:stirabos@redhat.com] 
Sent: Friday, December 19, 2014 12:28 AM 
To: Yue, Cong 
Cc: users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org> 
Subject: Re: [ovirt-users] VM failover with ovirt3.5
----- Original Message ----- 
From: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> 
To: users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org> 
Sent: Friday, December 19, 2014 2:14:33 AM 
Subject: [ovirt-users] VM failover with ovirt3.5
Hi
In my environment, I have 3 ovirt nodes as one cluster. And on top of 
host-1, there is one vm to host ovirt engine.
Also I have one external storage for the cluster to use as data domain 
of engine and data.
I confirmed live migration works well in my environment.
But it seems very buggy for VM failover if I try to force to shut down 
one ovirt node. Sometimes the VM in the node which is shutdown can 
migrate to other host, but it take more than several minutes.
Sometimes, it can not migrate at all. Sometimes, only when the host is 
back, the VM is beginning to move.
Can you please check or share the logs under /var/log/ovirt-hosted-engine-ha/ 
?
Is there some documentation to explain how VM failover is working? And 
is there some bugs reported related with this?
http://www.ovirt.org/Features/Self_Hosted_Engine#Agent_State_Diagram
Thanks in advance,
Cong
This e-mail message is for the sole use of the intended recipient(s) 
and may contain confidential and privileged information. Any 
unauthorized review, use, disclosure or distribution is prohibited. If 
you are not the intended recipient, please contact the sender by reply 
e-mail and destroy all copies of the original message. If you are the 
intended recipient, please be advised that the content of this message 
is subject to access, review and disclosure by the sender's e-mail System 
Administrator.
_______________________________________________ 
Users mailing list 
Users@ovirt.org<mailto:Users@ovirt.org><mailto:Users@ovirt.org> 
http://lists.ovirt.org/mailman/listinfo/users
This e-mail message is for the sole use of the intended recipient(s) and may 
contain confidential and privileged information. Any unauthorized review, 
use, disclosure or distribution is prohibited. If you are not the intended 
recipient, please contact the sender by reply e-mail and destroy all copies 
of the original message. If you are the intended recipient, please be 
advised that the content of this message is subject to access, review and 
disclosure by the sender's e-mail System Administrator.
This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. 
_______________________________________________ 
Users mailing list 
Users@ovirt.org<mailto:Users@ovirt.org><mailto:Users@ovirt.org> 
http://lists.ovirt.org/mailman/listinfo/users
________________________________ 
This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator.
________________________________ 
This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator.
...
Cong<br><div><br></div><br><div><br></div>> On 2014/12/29, at 8:43, "Ar=
tyom Lukianov" <alukiano@redhat.com> wrote:<br>><br>> I see tha=
t HE vm run on host with ip 10.0.0.94, and two another hosts in "Local Main=
tenance" state, so vm will not migrate to any of them, can you try disable =
local maintenance on all hosts in HE environment and after enable "local ma=
intenance" on host where HE vm run, and provide also output of hosted-engin=
e --vm-status.<br>> Failover works in next way:<br>> 1) if host where=
 run HE vm have score less by 800 that some other host in HE environment, H=
E vm will migrate on host with best score<br>> 2) if something happen to=
 vm(kernel panic, crash of service...), agent will restart HE vm on another=
 host in HE environment with positive score<br>> 3) if put to local main=
tenance host with HE vm, vm will migrate to another host with positive scor=
e<br>> Thanks.<br>><br>> ----- Original Message -----<br>> From=
: "Cong Yue" <Cong_Yue@alliedtelesis.com><br>> To: "Artyom Lukiano=
v" <alukiano@redhat.com><br>> Cc: "Simone Tiraboschi" <stirabos=
@redhat.com>, users@ovirt.org<br>> Sent: Monday, December 29, 2014 6:=
30:42 PM<br>> Subject: Re: [ovirt-users] VM failover with ovirt3.5<br>&g=
t;<br>> Thanks and the --vm-status log is as follows:<br>> [root@comp=
ute2-2 ~]# hosted-engine --vm-status<br>><br>><br>> --=3D=3D Host =
1 status =3D=3D--<br>><br>> Status up-to-date       &n=
bsp;          : True<br>> Hostname    =
;                     &nb=
sp; : 10.0.0.94<br>> Host ID             &=
nbsp;              : 1<br>> Engine st=
atus                     =
 : {"health": "good", "vm": "up",<br>> "detail": "up"}<br>> Scor=
e                     &nb=
sp;        : 2400<br>> Local maintenance   &nbs=
...
> score=3D2400<br>> maintenance=3DFalse<br>> state=3DEngineUp<br>=
><br>><br>> --=3D=3D Host 2 status =3D=3D--<br>><br>> Status=
 up-to-date                  :=
 True<br>> Hostname               &nb=
sp;           : 10.0.0.93<br>> Host ID   &=
nbsp;                    =
    : 2<br>> Engine status           =
           : {"reason": "vm not running on<br=
> this host", "health": "bad", "vm": "down", "detail": "unknown"}<br>&g=
t; Score                   &nb=
sp;          : 0<br>> Local maintenance   =
               : True<br>> Host =
timestamp                   &n=
bsp; : 859142<br>> Extra metadata (valid at timestamp):<br>> metadata=
_parse_version=3D1<br>> metadata_feature_version=3D1<br>> timestamp=
=3D859142 (Mon Dec 29 08:25:08 2014)<br>> host-id=3D2<br>> score=3D0<=
br>> maintenance=3DTrue<br>> state=3DLocalMaintenance<br>><br>>=
<br>> --=3D=3D Host 3 status =3D=3D--<br>><br>> Status up-to-date =
                 : True<br>>=
; Hostname                   &=
nbsp;       : 10.0.0.92<br>> Host ID      =
                     &nbs=
...
> MainThread::INFO::2014-12-28<br>> 20:49:37,961::hosted_engine::332=
::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitorin=
g)<br>> Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThre=
ad::INFO::2014-12-28<br>> 20:49:48,048::state_decorators::124::ovirt_hos=
ted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>> Local maint=
enance detected<br>> MainThread::INFO::2014-12-28<br>> 20:49:48,319::=
states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(scor=
e)<br>> Score is 0 due to local maintenance mode<br>> MainThread::INF=
O::2014-12-28<br>> 20:49:48,319::hosted_engine::327::ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current st=
ate LocalMaintenance (score: 0)<br>> MainThread::INFO::2014-12-28<br>>=
; 20:49:48,319::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_eng=
ine.HostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id=
: 1, score: 2400)<br>><br>> Thanks,<br>> Cong<br>><br>><br>&=
gt; On 2014/12/28, at 3:46, "Artyom Lukianov" <alukiano@redhat.com<ma=
ilto:alukiano@redhat.com><mailto:alukiano@redhat.com>> wrote:<b=
r>><br>> I see that you set local maintenance on host3 that do not ha=
ve engine vm on it, so it nothing to migrate from this host.<br>> If you=
 set local maintenance on host1, vm must migrate to another host with posit=
ive score.<br>> Thanks<br>><br>> ----- Original Message -----<br>&=
gt; From: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alli=
edtelesis.com><mailto:Cong_Yue@alliedtelesis.com>><br>> To: =
"Simone Tiraboschi" <stirabos@redhat.com<mailto:stirabos@redhat.com&g=
t;<mailto:stirabos@redhat.com>><br>> Cc: users@ovirt.org<mai=
lto:users@ovirt.org><mailto:users@ovirt.org><br>> Sent: Saturda=
y, December 27, 2014 6:58:32 PM<br>> Subject: Re: [ovirt-users] VM failo=
ver with ovirt3.5<br>><br>> Hi<br>><br>> I had a try with "host=
ed-engine --set-maintence --mode=3Dlocal" on<br>> compute2-1, which is h=
ost 3 in my cluster. From the log, it shows<br>> maintence mode is decte=
cted, but migration does not happen.<br>><br>> The logs are as follow=
s. Is there any other config I need to check?<br>><br>> [root@compute=
2-1 vdsm]# hosted-engine --vm-status<br>><br>><br>> --=3D=3D Host =
1 status =3D=3D-<br>><br>> Status up-to-date       &nb=
sp;          : True<br>> Hostname    =
                     &nbs=
...
> 11:36:49,449::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted=
_engine.HostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.93=
 (id: 2, score: 2400)<br>> MainThread::INFO::2014-12-27<br>> 11:36:59=
,739::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted=
Engine::(start_monitoring)<br>> Current state EngineUp (score: 2400)<br>=
> MainThread::INFO::2014-12-27<br>> 11:36:59,739::hosted_engine::332:=
:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring=
)<br>> Best remote host 10.0.0.93 (id: 2, score: 2400)<br>> MainThrea=
d::INFO::2014-12-27<br>> 11:37:09,779::states::394::ovirt_hosted_engine_=
ha.agent.hosted_engine.HostedEngine::(consume)<br>> Engine vm running on=
 localhost<br>> MainThread::INFO::2014-12-27<br>> 11:37:10,026::hoste=
d_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(st=
art_monitoring)<br>> Current state EngineUp (score: 2400)<br>> MainTh=
read::INFO::2014-12-27<br>> 11:37:10,026::hosted_engine::332::ovirt_host=
ed_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> B=
est remote host 10.0.0.93 (id: 2, score: 2400)<br>> MainThread::INFO::20=
14-12-27<br>> 11:37:20,331::hosted_engine::327::ovirt_hosted_engine_ha.a=
gent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state E=
ngineUp (score: 2400)<br>> MainThread::INFO::2014-12-27<br>> 11:37:20=
,331::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted=
Engine::(start_monitoring)<br>> Best remote host 10.0.0.93 (id: 2, score=
: 2400)<br>><br>><br>> [root@compute2-2 ~]# tail -f /var/log/ovirt=
-hosted-engine-ha/agent.log<br>> MainThread::INFO::2014-12-27<br>> 08=
:36:12,462::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.=
HostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id: 1,=
 score: 2400)<br>> MainThread::INFO::2014-12-27<br>> 08:36:22,797::ho=
sted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::=
(start_monitoring)<br>> Current state EngineDown (score: 2400)<br>> M=
ainThread::INFO::2014-12-27<br>> 08:36:22,798::hosted_engine::332::ovirt=
_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>&=
gt; Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThread::INF=
O::2014-12-27<br>> 08:36:32,876::states::437::ovirt_hosted_engine_ha.age=
nt.hosted_engine.HostedEngine::(consume)<br>> Engine vm is running on ho=
st 10.0.0.94 (id 1)<br>> MainThread::INFO::2014-12-27<br>> 08:36:33,1=
69::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEn=
gine::(start_monitoring)<br>> Current state EngineDown (score: 2400)<br>=
> MainThread::INFO::2014-12-27<br>> 08:36:33,169::hosted_engine::332:=
:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring=
)<br>> Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThrea=
d::INFO::2014-12-27<br>> 08:36:43,567::hosted_engine::327::ovirt_hosted_=
engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Curr=
ent state EngineDown (score: 2400)<br>> MainThread::INFO::2014-12-27<br>=
> 08:36:43,567::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_=
engine.HostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 =
(id: 1, score: 2400)<br>> MainThread::INFO::2014-12-27<br>> 08:36:53,=
858::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE=
ngine::(start_monitoring)<br>> Current state EngineDown (score: 2400)<br=
> MainThread::INFO::2014-12-27<br>> 08:36:53,858::hosted_engine::332=
::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitorin=
g)<br>> Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThre=
ad::INFO::2014-12-27<br>> 08:37:04,028::state_machine::160::ovirt_hosted=
_engine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>> Global metad=
ata: {'maintenance': False}<br>> MainThread::INFO::2014-12-27<br>> 08=
:37:04,028::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.=
HostedEngine::(refresh)<br>> Host 10.0.0.94 (id 1): {'extra':<br>> 'm=
etadata_parse_version=3D1\nmetadata_feature_version=3D1\ntimestamp=3D835987=
<br>> (Sat Dec 27 11:37:30<br>> 2014)\nhost-id=3D1\nscore=3D2400\nmai=
ntenance=3DFalse\nstate=3DEngineUp\n',<br>> 'hostname': '10.0.0.94', 'al=
ive': True, 'host-id': 1, 'engine-status':<br>> {'health': 'good', 'vm':=
 'up', 'detail': 'up'}, 'score': 2400,<br>> 'maintenance': False, 'host-=
ts': 835987}<br>> MainThread::INFO::2014-12-27<br>> 08:37:04,028::sta=
te_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(=
refresh)<br>> Host 10.0.0.92 (id 3): {'extra':<br>> 'metadata_parse_v=
ersion=3D1\nmetadata_feature_version=3D1\ntimestamp=3D681528<br>> (Sat D=
ec 27 08:37:41<br>> 2014)\nhost-id=3D3\nscore=3D0\nmaintenance=3DTrue\ns=
tate=3DLocalMaintenance\n',<br>> 'hostname': '10.0.0.92', 'alive': True,=
 'host-id': 3, 'engine-status':<br>> {'reason': 'vm not running on this =
host', 'health': 'bad', 'vm':<br>> 'down', 'detail': 'unknown'}, 'score'=
: 0, 'maintenance': True,<br>> 'host-ts': 681528}<br>> MainThread::IN=
FO::2014-12-27<br>> 08:37:04,028::state_machine::168::ovirt_hosted_engin=
e_ha.agent.hosted_engine.HostedEngine::(refresh)<br>> Local (id 2): {'en=
gine-health': {'reason': 'vm not running on this<br>> host', 'health': '=
bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge':<br>> True, 'mem-free=
': 15300.0, 'maintenance': False, 'cpu-load': 0.0215,<br>> 'gateway': Tr=
ue}<br>> MainThread::INFO::2014-12-27<br>> 08:37:04,265::hosted_engin=
e::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_mon=
itoring)<br>> Current state EngineDown (score: 2400)<br>> MainThread:=
:INFO::2014-12-27<br>> 08:37:04,265::hosted_engine::332::ovirt_hosted_en=
gine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Best r=
emote host 10.0.0.94 (id: 1, score: 2400)<br>><br>> Thanks,<br>> C=
ong<br>><br>> On 2014/12/22, at 5:29, "Simone Tiraboschi" <stirabo=
s@redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.co=
m>> wrote:<br>><br>><br>><br>> ----- Original Message ---=
--<br>> From: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Y=
ue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>><br>&g=
t; To: "Simone Tiraboschi" <stirabos@redhat.com<mailto:stirabos@redha=
t.com><mailto:stirabos@redhat.com>><br>> Cc: users@ovirt.org=
<mailto:users@ovirt.org><mailto:users@ovirt.org><br>> Sent: =
Friday, December 19, 2014 7:22:10 PM<br>> Subject: RE: [ovirt-users] VM =
failover with ovirt3.5<br>><br>> Thanks for the information. This is =
...
<div><br></div>This e-mail message is for the sole use of the intended rec=
ipient(s) and may contain confidential and privileged information. Any unau=
This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. 

------------------------------ 

_______________________________________________ 
Users mailing list 
Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users 

End of Users Digest, Vol 39, Issue 169 
************************************** 

------=_Part_1875460_365779577.1419876418683
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: georgia,serif; font-size: 12pt; colo=
r: #000000"><div>Hi,</div><div>Your guest-vm have to be defined as "<span s=
tyle=3D"font-family: 'Arial Unicode MS', Arial, sans-serif; font-size: smal=
l; line-height: 16px; background-color: #ffffff;" data-mce-style=3D"font-fa=
mily: 'Arial Unicode MS', Arial, sans-serif; font-size: small; line-height:=
 16px; background-color: #ffffff;">Highly Available"</span></div><div><tabl=
e xmlns:d=3D"http://docbook.org/ns/docbook" class=3D"lt-4-cols lt-7-rows mc=
eItemTable" summary=3D"Virtual Machine: High Availability Settings" style=
=3D"widows: 4; orphans: 4; border: 1px solid #aaaaaa; width: 768.7999877929=
688px; border-collapse: collapse; table-layout: fixed; word-wrap: break-wor=
d; color: #000000; font-family: 'liberation sans', 'Myriad ', 'Bitstream Ve=
ra Sans', 'Lucida Grande', 'Luxi Sans', 'Trebuchet MS', helvetica, verdana,=
 arial, sans-serif; font-size: 14.399999618530273px; line-height: 18.049999=
237060547px; background-color: #ffffff;" data-mce-style=3D"widows: 4; orpha=
ns: 4; border: 1px solid #aaaaaa; width: 768.7999877929688px; border-collap=
se: collapse; table-layout: fixed; word-wrap: break-word; color: #000000; f=
ont-family: 'liberation sans', 'Myriad ', 'Bitstream Vera Sans', 'Lucida Gr=
ande', 'Luxi Sans', 'Trebuchet MS', helvetica, verdana, arial, sans-serif; =
font-size: 14.399999618530273px; line-height: 18.049999237060547px; backgro=
und-color: #ffffff;"><tbody><tr><td align=3D"left" style=3D"border: none; v=
ertical-align: top; padding: 0.15em 0.5em;" data-mce-style=3D"border: none;=
 vertical-align: top; padding: 0.15em 0.5em;"><div class=3D"para" style=3D"=
line-height: 1.29em; padding-top: 0px; margin-top: 0px; padding-bottom: 0px=
; margin-bottom: 1em; display: inline;" data-mce-style=3D"line-height: 1.29=
em; padding-top: 0px; margin-top: 0px; padding-bottom: 0px; margin-bottom: =
1em; display: inline;"><span class=3D"guilabel" style=3D"font-family: 'deja=
vu sans mono', 'liberation mono', 'bitstream vera mono', 'dejavu mono', mon=
ospace; font-weight: bold;" data-mce-style=3D"font-family: 'dejavu sans mon=
o', 'liberation mono', 'bitstream vera mono', 'dejavu mono', monospace; fon=
t-weight: bold;"><strong>Highly Available</strong></span></div></td><td ali=
gn=3D"left" style=3D"border: none; vertical-align: top; padding: 0.15em 0.5=
em;" data-mce-style=3D"border: none; vertical-align: top; padding: 0.15em 0=
.5em;"><div class=3D"para" style=3D"line-height: 1.29em; padding-top: 0px; =
margin-top: 0px; padding-bottom: 0px; margin-bottom: 1em; display: inline;"=
 data-mce-style=3D"line-height: 1.29em; padding-top: 0px; margin-top: 0px; =
padding-bottom: 0px; margin-bottom: 1em; display: inline;">Select this chec=
k box if the virtual machine is to be highly available. For example, in cas=
es of host maintenance or failure, the virtual machine is automatically mov=
ed to or re-launched on another host. If the host is manually shut down by =
the system administrator, the virtual machine is not automatically moved to=
 another host.</div><div class=3D"para" style=3D"line-height: 1.29em; paddi=
ng-top: 0px; margin-top: 1em; padding-bottom: 0px; margin-bottom: 1em;" dat=
a-mce-style=3D"line-height: 1.29em; padding-top: 0px; margin-top: 1em; padd=
ing-bottom: 0px; margin-bottom: 1em;">Note that this option is unavailable =
if the <span class=3D"guilabel" style=3D"font-family: 'dejavu sans mon=
o', 'liberation mono', 'bitstream vera mono', 'dejavu mono', monospace; fon=
t-weight: bold;" data-mce-style=3D"font-family: 'dejavu sans mono', 'libera=
tion mono', 'bitstream vera mono', 'dejavu mono', monospace; font-weight: b=
old;">Migration Options</span> setting in the <span class=3D"guil=
abel" style=3D"font-family: 'dejavu sans mono', 'liberation mono', 'bitstre=
am vera mono', 'dejavu mono', monospace; font-weight: bold;" data-mce-style=
=3D"font-family: 'dejavu sans mono', 'liberation mono', 'bitstream vera mon=
o', 'dejavu mono', monospace; font-weight: bold;">Hosts</span> tab is =
set to either <span class=3D"guilabel" style=3D"font-family: 'dejavu s=
ans mono', 'liberation mono', 'bitstream vera mono', 'dejavu mono', monospa=
ce; font-weight: bold;" data-mce-style=3D"font-family: 'dejavu sans mono', =
'liberation mono', 'bitstream vera mono', 'dejavu mono', monospace; font-we=
ight: bold;">Allow manual migration only</span> or <span class=3D=
"guilabel" style=3D"font-family: 'dejavu sans mono', 'liberation mono', 'bi=
tstream vera mono', 'dejavu mono', monospace; font-weight: bold;" data-mce-=
style=3D"font-family: 'dejavu sans mono', 'liberation mono', 'bitstream ver=
a mono', 'dejavu mono', monospace; font-weight: bold;">No migration</span>.=
 For a virtual machine to be highly available, it must be possible for the =
Manager to migrate the virtual machine to other available hosts as necessar=
y.</div></td></tr></tbody></table><div><br></div></div><div><span name=3D"x=
"></span><br>Thanks in advance.<br><div><br></div>Best regards,<br>Nikolai<=
br>____________________<br>Nikolai Sednev<br>Senior Quality Engineer at Com=
pute team<br>Red Hat Israel<br>34 Jerusalem Road,<br>Ra'anana, Israel 43501=
<br><div><br></div>Tel:       +972   9 7692043<br>Mobil=
e: +972 52 7342734<br>Email: nsednev@redhat.com<br>IRC: nsednev<span name=
=3D"x"></span><br></div><div><br></div><hr id=3D"zwchr"><div style=3D"color=
:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family=
:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>users-request@ovi=
rt.org<br><b>To: </b>users@ovirt.org<br><b>Sent: </b>Monday, December 29, 2=
014 7:50:07 PM<br><b>Subject: </b>Users Digest, Vol 39, Issue 169<br><div><=
br></div>Send Users mailing list submissions to<br>    =
    users@ovirt.org<br><div><br></div>To subscribe or u=
nsubscribe via the World Wide Web, visit<br>     &=
nbsp;  http://lists.ovirt.org/mailman/listinfo/users<br>or, via e=
mail, send a message with subject or body 'help' to<br>   &n=
bsp;    users-request@ovirt.org<br><div><br></div>You c=
an reach the person managing the list at<br>     &=
nbsp;  users-owner@ovirt.org<br><div><br></div>When replying, ple=
ase edit your Subject line so it is more specific<br>than "Re: Contents of =
Users digest..."<br><div><br></div><br>Today's Topics:<br><div><br></div>&n=
bsp;  1. Re:  VM failover with ovirt3.5 (Yue, Cong)<br><div><br><=
/div><br>------------------------------------------------------------------=
----<br><div><br></div>Message: 1<br>Date: Mon, 29 Dec 2014 09:49:58 -0800<=
br>From: "Yue, Cong" <Cong_Yue@alliedtelesis.com><br>To: Artyom Lukia=
nov <alukiano@redhat.com><br>Cc: "users@ovirt.org" <users@ovirt.or=
g><br>Subject: Re: [ovirt-users] VM failover with ovirt3.5<br>Message-ID=
: <11A51118-8B03-41FE-8FD0-C81AC8897EF6@alliedtelesis.com><br>Content=
-Type: text/plain; charset=3D"us-ascii"<br><div><br></div>Thanks for detail=
ed explanation. Do you mean only HE VM can be failover? I want to have a tr=
y with the VM on any host to check whether VM can be failover to other host=
 automatically like VMware or Xenserver?<br>I will have a try as you advise=
d and provide the log for your further advice.<br><div><br></div>Thanks,<br=
p;              : False<br>> Host tim=
estamp                    =
; : 1008087<br>> Extra metadata (valid at timestamp):<br>> metadata_p=
arse_version=3D1<br>> metadata_feature_version=3D1<br>> timestamp=3D1=
008087<tel:1008087> (Mon Dec 29 11:25:51 2014)<br>> host-id=3D1<br=
p;: 3<br>> Engine status              =
;        : {"reason": "vm not running on<br>> this h=
ost", "health": "bad", "vm": "down", "detail": "unknown"}<br>> Score &nb=
sp;                     &=
nbsp;      : 0<br>> Local maintenance      =
;            : True<br>> Host timestamp &n=
bsp;                   : 85361=
5<br>> Extra metadata (valid at timestamp):<br>> metadata_parse_versi=
on=3D1<br>> metadata_feature_version=3D1<br>> timestamp=3D853615 (Mon=
 Dec 29 08:25:57 2014)<br>> host-id=3D3<br>> score=3D0<br>> mainte=
nance=3DTrue<br>> state=3DLocalMaintenance<br>> You have new mail in =
/var/spool/mail/root<br>> [root@compute2-2 ~]#<br>><br>> Could you=
 please explain how VM failover works inside ovirt? Is there any other debu=
g option I can enable to check the problem?<br>><br>> Thanks,<br>>=
 Cong<br>><br>><br>> On 2014/12/29, at 1:39, "Artyom Lukianov" <=
;alukiano@redhat.com<mailto:alukiano@redhat.com>> wrote:<br>><b=
r>> Can you also provide output of hosted-engine --vm-status please, pre=
vious time it was useful, because I do not see something unusual.<br>> T=
hanks<br>><br>> ----- Original Message -----<br>> From: "Cong Yue"=
 <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com>>=
;<br>> To: "Artyom Lukianov" <alukiano@redhat.com<mailto:alukiano@=
redhat.com>><br>> Cc: "Simone Tiraboschi" <stirabos@redhat.com&=
lt;mailto:stirabos@redhat.com>>, users@ovirt.org<mailto:users@ovir=
t.org><br>> Sent: Monday, December 29, 2014 7:15:24 AM<br>> Subjec=
t: Re: [ovirt-users] VM failover with ovirt3.5<br>><br>> Also I chang=
e the maintenance mode to local in another host. But also the VM in this ho=
st can not be migrated. The logs are as follows.<br>><br>> [root@comp=
ute2-2 ~]# hosted-engine --set-maintenance --mode=3Dlocal<br>> [root@com=
pute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log<br>> MainT=
hread::INFO::2014-12-28<br>> 21:09:04,184::hosted_engine::332::ovirt_hos=
ted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> =
Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThread::INFO::2=
014-12-28<br>> 21:09:14,603::hosted_engine::327::ovirt_hosted_engine_ha.=
agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state =
EngineDown (score: 2400)<br>> MainThread::INFO::2014-12-28<br>> 21:09=
:14,603::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hos=
tedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id: 1, sc=
ore: 2400)<br>> MainThread::INFO::2014-12-28<br>> 21:09:24,903::hoste=
d_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(st=
art_monitoring)<br>> Current state EngineDown (score: 2400)<br>> Main=
Thread::INFO::2014-12-28<br>> 21:09:24,904::hosted_engine::332::ovirt_ho=
sted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>>=
 Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThread::INFO::=
2014-12-28<br>> 21:09:35,026::states::437::ovirt_hosted_engine_ha.agent.=
hosted_engine.HostedEngine::(consume)<br>> Engine vm is running on host =
10.0.0.94 (id 1)<br>> MainThread::INFO::2014-12-28<br>> 21:09:35,236:=
:hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin=
e::(start_monitoring)<br>> Current state EngineDown (score: 2400)<br>>=
; MainThread::INFO::2014-12-28<br>> 21:09:35,236::hosted_engine::332::ov=
irt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<b=
r>> Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThread::=
INFO::2014-12-28<br>> 21:09:45,604::hosted_engine::327::ovirt_hosted_eng=
ine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current=
 state EngineDown (score: 2400)<br>> MainThread::INFO::2014-12-28<br>>=
; 21:09:45,604::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_eng=
ine.HostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id=
: 1, score: 2400)<br>> MainThread::INFO::2014-12-28<br>> 21:09:55,691=
::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE=
ngine::(check)<br>> Local maintenance detected<br>> MainThread::INFO:=
:2014-12-28<br>> 21:09:55,701::brokerlink::111::ovirt_hosted_engine_ha.l=
ib.brokerlink.BrokerLink::(notify)<br>> Trying: notify time=3D1419829795=
.7 type=3Dstate_transition<br>> detail=3DEngineDown-LocalMaintenance hos=
tname=3D'compute2-2'<br>> MainThread::INFO::2014-12-28<br>> 21:09:55,=
761::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(no=
tify)<br>> Success, was notification of state_transition<br>> (Engine=
Down-LocalMaintenance) sent? sent<br>> MainThread::INFO::2014-12-28<br>&=
gt; 21:09:55,990::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.H=
ostedEngine::(score)<br>> Score is 0 due to local maintenance mode<br>&g=
t; MainThread::INFO::2014-12-28<br>> 21:09:55,990::hosted_engine::327::o=
virt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<=
br>> Current state LocalMaintenance (score: 0)<br>> MainThread::INFO:=
:2014-12-28<br>> 21:09:55,991::hosted_engine::332::ovirt_hosted_engine_h=
a.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Best remote =
host 10.0.0.94 (id: 1, score: 2400)<br>> ^C<br>> You have new mail in=
 /var/spool/mail/root<br>> [root@compute2-2 ~]# ps -ef | grep qemu<br>&g=
t; root     18420  2777  0 21:10<x-apple-data-detect=
ors://39> pts/0    00:00:00<x-apple-data-detectors://40>=
 grep --color=3Dauto qemu<br>> qemu     29809     1 =
 0 Dec19 ?        01:17:20 /usr/libexec/qemu-kvm<b=
r>> -name testvm2-2 -S -machine rhel6.5.0,accel=3Dkvm,usb=3Doff -cpu Neh=
alem<br>> -m 500 -realtime mlock=3Doff -smp<br>> 1,maxcpus=3D16,socke=
ts=3D16,cores=3D1,threads=3D1 -uuid<br>> c31e97d0-135e-42da-9954-162b522=
8dce3 -smbios<br>> type=3D1,manufacturer=3DoVirt,product=3DoVirt<br>>=
 Node,version=3D7-0.1406.el7.centos.2.5,serial=3D4C4C4544-0059-3610-8033-B4=
C04F395931,uuid=3Dc31e97d0-135e-42da-9954-162b5228dce3<br>> -no-user-con=
fig -nodefaults -chardev<br>> socket,id=3Dcharmonitor,path=3D/var/lib/li=
bvirt/qemu/testvm2-2.monitor,server,nowait<br>> -mon chardev=3Dcharmonit=
or,id=3Dmonitor,mode=3Dcontrol -rtc<br>> base=3D2014-12-19T20:17:17<x=
-apple-data-detectors://42>,driftfix=3Dslew -no-kvm-pit-reinjection<br>&=
gt; -no-hpet -no-shutdown -boot strict=3Don -device<br>> piix3-usb-uhci,=
id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -device<br>> virtio-scsi-pci,id=3Dsc=
si0,bus=3Dpci.0,addr=3D0x4 -device<br>> virtio-serial-pci,id=3Dvirtio-se=
rial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x5<br>> -drive if=3Dnone,id=3Dd=
rive-ide0-1-0,readonly=3Don,format=3Draw,serial=3D<br>> -device ide-cd,b=
us=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0<br>> -drive fil=
e=3D/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096-27c4-42=
56-b2ac-bd7265525c69/images/5cbeb8c9-4f04-48d0-a5eb-78c49187c550/a0570e8c-9=
867-4ec4-818f-11e102fc4f9b,if=3Dnone,id=3Ddrive-virtio-disk0,format=3Dqcow2=
,serial=3D5cbeb8c9-4f04-48d0-a5eb-78c49187c550,cache=3Dnone,werror=3Dstop,r=
error=3Dstop,aio=3Dthreads<br>> -device virtio-blk-pci,scsi=3Doff,bus=3D=
pci.0,addr=3D0x6,drive=3Ddrive-virtio-disk0,id=3Dvirtio-disk0,bootindex=3D1=
<br>> -netdev tap,fd=3D28,id=3Dhostnet0,vhost=3Don,vhostfd=3D29 -device<=
br>> virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:db:94:00,=
bus=3Dpci.0,addr=3D0x3<br>> -chardev socket,id=3Dcharchannel0,path=3D/va=
r/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3.com.redhat=
.rhevm.vdsm,server,nowait<br>> -device virtserialport,bus=3Dvirtio-seria=
l0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dchannel0,name=3Dcom.redhat.rhevm.vd=
sm<br>> -chardev socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/c=
hannels/c31e97d0-135e-42da-9954-162b5228dce3.org.qemu.guest_agent.0,server,=
nowait<br>> -device virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=
=3Dcharchannel1,id=3Dchannel1,name=3Dorg.qemu.guest_agent.0<br>> -charde=
v spicevmc,id=3Dcharchannel2,name=3Dvdagent -device<br>> virtserialport,=
bus=3Dvirtio-serial0.0,nr=3D3,chardev=3Dcharchannel2,id=3Dchannel2,name=3Dc=
om.redhat.spice.0<br>> -spice tls-port=3D5901,addr=3D10.0.0.93,x509-dir=
=3D/etc/pki/vdsm/libvirt-spice,tls-channel=3Dmain,tls-channel=3Ddisplay,tls=
-channel=3Dinputs,tls-channel=3Dcursor,tls-channel=3Dplayback,tls-channel=
=3Drecord,tls-channel=3Dsmartcard,tls-channel=3Dusbredir,seamless-migration=
=3Don<br>> -k en-us -vga qxl -global qxl-vga.ram_size=3D67108864<tel:=
67108864> -global<br>> qxl-vga.vram_size=3D33554432<tel:33554432&g=
t; -incoming tcp:[::]:49152 -device<br>> virtio-balloon-pci,id=3Dballoon=
0,bus=3Dpci.0,addr=3D0x7<br>> [root@compute2-2 ~]#<br>><br>> Thank=
s,<br>> Cong<br>><br>><br>> On 2014/12/28, at 20:53, "Yue, Cong=
" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com>&l=
t;mailto:Cong_Yue@alliedtelesis.com>> wrote:<br>><br>> I checke=
d it again and confirmed there is one guest VM is running on the top of thi=
s host. The log is as follows:<br>><br>> [root@compute2-1 vdsm]# ps -=
ef | grep qemu<br>> qemu      2983   846  0 Dec=
19 ?        00:00:00<x-apple-data-detectors://0> =
[supervdsmServer] <defunct><br>> root      5489 &nb=
sp;3053  0 20:49<x-apple-data-detectors://1> pts/0    =
00:00:00<x-apple-data-detectors://2> grep --color=3Dauto qemu<br>>=
 qemu     26128     1  0 Dec19 ?     &nb=
sp;  01:09:19 /usr/libexec/qemu-kvm<br>> -name testvm2 -S -machine =
rhel6.5.0,accel=3Dkvm,usb=3Doff -cpu Nehalem -m<br>> 500 -realtime mlock=
=3Doff -smp 1,maxcpus=3D16,sockets=3D16,cores=3D1,threads=3D1<br>> -uuid=
 e46bca87-4df5-4287-844b-90a26fccef33 -smbios<br>> type=3D1,manufacturer=
=3DoVirt,product=3DoVirt<br>> Node,version=3D7-0.1406.el7.centos.2.5,ser=
ial=3D4C4C4544-0030-3310-8059-B8C04F585231,uuid=3De46bca87-4df5-4287-844b-9=
0a26fccef33<br>> -no-user-config -nodefaults -chardev<br>> socket,id=
=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/testvm2.monitor,server,nowait<b=
r>> -mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol -rtc<br>> =
base=3D2014-12-19T20:18:01<x-apple-data-detectors://4>,driftfix=3Dsle=
w -no-kvm-pit-reinjection<br>> -no-hpet -no-shutdown -boot strict=3Don -=
device<br>> piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -device<b=
r>> virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4 -device<br>> vi=
rtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x5<b=
r>> -drive if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don,format=3Draw,seri=
al=3D<br>> -device ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=
=3Dide0-1-0<br>> -drive file=3D/rhev/data-center/00000002-0002-0002-0002=
-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/b4b5426b-95e3-41a=
f-b286-da245891cdaf/0f688d49-97e3-4f1d-84d4-ac1432d903b3,if=3Dnone,id=3Ddri=
ve-virtio-disk0,format=3Dqcow2,serial=3Db4b5426b-95e3-41af-b286-da245891cda=
f,cache=3Dnone,werror=3Dstop,rerror=3Dstop,aio=3Dthreads<br>> -device vi=
rtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk0,i=
d=3Dvirtio-disk0,bootindex=3D1<br>> -netdev tap,fd=3D26,id=3Dhostnet0,vh=
ost=3Don,vhostfd=3D27 -device<br>> virtio-net-pci,netdev=3Dhostnet0,id=
=3Dnet0,mac=3D00:1a:4a:db:94:01,bus=3Dpci.0,addr=3D0x3<br>> -chardev soc=
ket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/e46bca87-4df5-4=
287-844b-90a26fccef33.com.redhat.rhevm.vdsm,server,nowait<br>> -device v=
irtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dcha=
nnel0,name=3Dcom.redhat.rhevm.vdsm<br>> -chardev socket,id=3Dcharchannel=
1,path=3D/var/lib/libvirt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef3=
3.org.qemu.guest_agent.0,server,nowait<br>> -device virtserialport,bus=
=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dchannel1,name=3Dorg.=
qemu.guest_agent.0<br>> -chardev spicevmc,id=3Dcharchannel2,name=3Dvdage=
nt -device<br>> virtserialport,bus=3Dvirtio-serial0.0,nr=3D3,chardev=3Dc=
harchannel2,id=3Dchannel2,name=3Dcom.redhat.spice.0<br>> -spice tls-port=
=3D5900,addr=3D10.0.0.92,x509-dir=3D/etc/pki/vdsm/libvirt-spice,tls-channel=
=3Dmain,tls-channel=3Ddisplay,tls-channel=3Dinputs,tls-channel=3Dcursor,tls=
-channel=3Dplayback,tls-channel=3Drecord,tls-channel=3Dsmartcard,tls-channe=
l=3Dusbredir,seamless-migration=3Don<br>> -k en-us -vga qxl -global qxl-=
vga.ram_size=3D67108864<tel:67108864> -global<br>> qxl-vga.vram_si=
ze=3D33554432<tel:33554432> -incoming tcp:[::]:49152 -device<br>> =
virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7<br>> [root@compu=
te2-1 vdsm]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log<br>> Main=
Thread::INFO::2014-12-28<br>> 20:49:27,315::state_decorators::124::ovirt=
_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>> Local m=
aintenance detected<br>> MainThread::INFO::2014-12-28<br>> 20:49:27,6=
46::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEn=
gine::(start_monitoring)<br>> Current state LocalMaintenance (score: 0)<=
br>> MainThread::INFO::2014-12-28<br>> 20:49:27,646::hosted_engine::3=
32::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitor=
ing)<br>> Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainTh=
read::INFO::2014-12-28<br>> 20:49:37,732::state_decorators::124::ovirt_h=
osted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>> Local mai=
ntenance detected<br>> MainThread::INFO::2014-12-28<br>> 20:49:37,961=
::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngi=
ne::(start_monitoring)<br>> Current state LocalMaintenance (score: 0)<br=
p; : 10.0.0.94<br>> Host ID             &n=
bsp;              : 1<br>> Engine sta=
tus                     &=
nbsp;: {"health": "good", "vm": "up",<br>> "detail": "up"}<br>> Score=
                     &nbs=
p;        : 2400<br>> Local maintenance    =
;              : False<br>> Host time=
stamp                    =
 : 836296<br>> Extra metadata (valid at timestamp):<br>> metadata_par=
se_version=3D1<br>> metadata_feature_version=3D1<br>> timestamp=3D836=
296 (Sat Dec 27 11:42:39 2014)<br>> host-id=3D1<br>> score=3D2400<br>=
> maintenance=3DFalse<br>> state=3DEngineUp<br>><br>><br>> -=
-=3D=3D Host 2 status =3D=3D--<br>><br>> Status up-to-date   &nb=
sp;              : True<br>> Hostname=
                     &nbs=
p;     : 10.0.0.93<br>> Host ID         &n=
bsp;                  : 2<br>&=
gt; Engine status                 &=
nbsp;    : {"reason": "vm not running on<br>> this host", "hea=
lth": "bad", "vm": "down", "detail": "unknown"}<br>> Score    =
                     &nbs=
p;    : 2400<br>> Local maintenance        =
;          : False<br>> Host timestamp   &=
nbsp;                 : 687358<br>&=
gt; Extra metadata (valid at timestamp):<br>> metadata_parse_version=3D1=
<br>> metadata_feature_version=3D1<br>> timestamp=3D687358 (Sat Dec 2=
7 08:42:04 2014)<br>> host-id=3D2<br>> score=3D2400<br>> maintenan=
ce=3DFalse<br>> state=3DEngineDown<br>><br>><br>> --=3D=3D Host=
 3 status =3D=3D--<br>><br>> Status up-to-date       &=
nbsp;          : True<br>> Hostname   &nbs=
p;                     &n=
bsp; : 10.0.0.92<br>> Host ID             =
               : 3<br>> Engine s=
tatus                    =
  : {"reason": "vm not running on<br>> this host", "health": "bad",=
 "vm": "down", "detail": "unknown"}<br>> Score       &nbs=
p;                     &n=
bsp;: 0<br>> Local maintenance            =
      : True<br>> Host timestamp       &nb=
sp;             : 681827<br>> Extra metada=
ta (valid at timestamp):<br>> metadata_parse_version=3D1<br>> metadat=
a_feature_version=3D1<br>> timestamp=3D681827 (Sat Dec 27 08:42:40 2014)=
<br>> host-id=3D3<br>> score=3D0<br>> maintenance=3DTrue<br>> s=
tate=3DLocalMaintenance<br>> [root@compute2-1 vdsm]# tail -f /var/log/ov=
irt-hosted-engine-ha/agent.log<br>> MainThread::INFO::2014-12-27<br>>=
 08:42:41,109::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engi=
ne.HostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id:=
 1, score: 2400)<br>> MainThread::INFO::2014-12-27<br>> 08:42:51,198:=
:state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEn=
gine::(check)<br>> Local maintenance detected<br>> MainThread::INFO::=
2014-12-27<br>> 08:42:51,420::hosted_engine::327::ovirt_hosted_engine_ha=
.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state=
 LocalMaintenance (score: 0)<br>> MainThread::INFO::2014-12-27<br>> 0=
8:42:51,420::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine=
.HostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id: 1=
, score: 2400)<br>> MainThread::INFO::2014-12-27<br>> 08:43:01,507::s=
tate_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngi=
ne::(check)<br>> Local maintenance detected<br>> MainThread::INFO::20=
14-12-27<br>> 08:43:01,773::hosted_engine::327::ovirt_hosted_engine_ha.a=
gent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state L=
ocalMaintenance (score: 0)<br>> MainThread::INFO::2014-12-27<br>> 08:=
43:01,773::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.H=
ostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id: 1, =
score: 2400)<br>> MainThread::INFO::2014-12-27<br>> 08:43:11,859::sta=
te_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine=
::(check)<br>> Local maintenance detected<br>> MainThread::INFO::2014=
-12-27<br>> 08:43:12,072::hosted_engine::327::ovirt_hosted_engine_ha.age=
nt.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state Loc=
alMaintenance (score: 0)<br>> MainThread::INFO::2014-12-27<br>> 08:43=
:12,072::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hos=
tedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id: 1, sc=
ore: 2400)<br>><br>><br>><br>> [root@compute2-3 ~]# tail -f /va=
r/log/ovirt-hosted-engine-ha/agent.log<br>> MainThread::INFO::2014-12-27=
<br>> 11:36:28,855::hosted_engine::332::ovirt_hosted_engine_ha.agent.hos=
ted_engine.HostedEngine::(start_monitoring)<br>> Best remote host 10.0.0=
.93 (id: 2, score: 2400)<br>> MainThread::INFO::2014-12-27<br>> 11:36=
:39,130::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.Hos=
tedEngine::(start_monitoring)<br>> Current state EngineUp (score: 2400)<=
br>> MainThread::INFO::2014-12-27<br>> 11:36:39,130::hosted_engine::3=
32::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitor=
ing)<br>> Best remote host 10.0.0.93 (id: 2, score: 2400)<br>> MainTh=
read::INFO::2014-12-27<br>> 11:36:49,449::hosted_engine::327::ovirt_host=
ed_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> C=
urrent state EngineUp (score: 2400)<br>> MainThread::INFO::2014-12-27<br=
the log for my three ovirt nodes.<br>> From the output of hosted-engine =
--vm-status, it shows the engine state for<br>> my 2nd and 3rd ovirt nod=
e is DOWN.<br>> Is this the reason why VM failover not work in my enviro=
nment?<br>><br>> No, they looks ok: you can run the engine VM on sing=
le host at a time.<br>><br>> How can I make<br>> also engine works=
 for my 2nd and 3rd ovit nodes?<br>><br>> If you put the host 1 in lo=
cal maintenance mode ( hosted-engine --set-maintenance --mode=3Dlocal ) the=
 VM should migrate to host 2; if you reactivate host 1 ( hosted-engine --se=
t-maintenance --mode=3Dnone ) and put host 2 in local maintenance mode the =
VM should migrate again.<br>><br>> Can you please try that and post t=
he logs if something is going bad?<br>><br>><br>> --<br>> --=3D=
=3D Host 1 status =3D=3D--<br>><br>> Status up-to-date     =
             : True<br>> Hostname &nb=
sp;                     &=
nbsp;   : 10.0.0.94<br>> Host ID          =
                  : 1<br>> =
Engine status                  =
;    : {"health": "good", "vm": "up",<br>> "detail": "up"}<br>=
> Score                   &=
nbsp;          : 2400<br>> Local maintenance &n=
bsp;                : False<br>>=
 Host timestamp                 &nb=
sp;   : 150475<br>> Extra metadata (valid at timestamp):<br>> me=
tadata_parse_version=3D1<br>> metadata_feature_version=3D1<br>> times=
tamp=3D150475 (Fri Dec 19 13:12:18 2014)<br>> host-id=3D1<br>> score=
=3D2400<br>> maintenance=3DFalse<br>> state=3DEngineUp<br>><br>>=
;<br>> --=3D=3D Host 2 status =3D=3D--<br>><br>> Status up-to-date=
                  : True<br>&g=
t; Hostname                   =
        : 10.0.0.93<br>> Host ID      =
;                     &nb=
sp;: 2<br>> Engine status             &nbs=
p;        : {"reason": "vm not running on<br>> this =
host", "health": "bad", "vm": "down", "detail": "unknown"}<br>> Score &n=
bsp;                     =
       : 2400<br>> Local maintenance     &=
nbsp;            : False<br>> Host timesta=
mp                     : =
1572<br>> Extra metadata (valid at timestamp):<br>> metadata_parse_ve=
rsion=3D1<br>> metadata_feature_version=3D1<br>> timestamp=3D1572 (Fr=
i Dec 19 10:12:18 2014)<br>> host-id=3D2<br>> score=3D2400<br>> ma=
intenance=3DFalse<br>> state=3DEngineDown<br>><br>><br>> --=3D=
=3D Host 3 status =3D=3D--<br>><br>> Status up-to-date     =
             : False<br>> Hostname &n=
bsp;                     =
    : 10.0.0.92<br>> Host ID          =
;                  : 3<br>>=
 Engine status                 &nbs=
p;    : unknown stale-data<br>> Score       &nb=
sp;                     &=
nbsp;: 2400<br>> Local maintenance           &n=
bsp;      : False<br>> Host timestamp      =
;               : 987<br>> Extra meta=
data (valid at timestamp):<br>> metadata_parse_version=3D1<br>> metad=
ata_feature_version=3D1<br>> timestamp=3D987 (Fri Dec 19 10:09:58 2014)<=
br>> host-id=3D3<br>> score=3D2400<br>> maintenance=3DFalse<br>>=
; state=3DEngineDown<br>><br>> --<br>> And the /var/log/ovirt-host=
ed-engine-ha/agent.log for three ovirt nodes are<br>> as follows:<br>>=
; --<br>> 10.0.0.94(hosted-engine-1)<br>> ---<br>> MainThread::INF=
O::2014-12-19<br>> 13:09:33,716::hosted_engine::327::ovirt_hosted_engine=
_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current st=
ate EngineUp (score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:=
09:33,716::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.H=
ostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.93 (id: 2, =
score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:09:44,017::hos=
ted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(=
start_monitoring)<br>> Current state EngineUp (score: 2400)<br>> Main=
Thread::INFO::2014-12-19<br>> 13:09:44,017::hosted_engine::332::ovirt_ho=
sted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>>=
 Best remote host 10.0.0.93 (id: 2, score: 2400)<br>> MainThread::INFO::=
2014-12-19<br>> 13:09:54,303::hosted_engine::327::ovirt_hosted_engine_ha=
.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state=
 EngineUp (score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:09:=
54,303::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Host=
edEngine::(start_monitoring)<br>> Best remote host 10.0.0.93 (id: 2, sco=
re: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:10:04,342::states=
::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)<b=
r>> Engine vm running on localhost<br>> MainThread::INFO::2014-12-19<=
br>> 13:10:04,617::hosted_engine::327::ovirt_hosted_engine_ha.agent.host=
ed_engine.HostedEngine::(start_monitoring)<br>> Current state EngineUp (=
score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:10:04,617::hos=
ted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(=
start_monitoring)<br>> Best remote host 10.0.0.93 (id: 2, score: 2400)<b=
r>> MainThread::INFO::2014-12-19<br>> 13:10:14,657::state_machine::16=
0::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>&g=
t; Global metadata: {'maintenance': False}<br>> MainThread::INFO::2014-1=
2-19<br>> 13:10:14,657::state_machine::165::ovirt_hosted_engine_ha.agent=
.hosted_engine.HostedEngine::(refresh)<br>> Host 10.0.0.93 (id 2): {'ext=
ra':<br>> 'metadata_parse_version=3D1\nmetadata_feature_version=3D1\ntim=
estamp=3D1448<br>> (Fri Dec 19 10:10:14<br>> 2014)\nhost-id=3D2\nscor=
e=3D2400\nmaintenance=3DFalse\nstate=3DEngineDown\n',<br>> 'hostname': '=
10.0.0.93', 'alive': True, 'host-id': 2, 'engine-status':<br>> {'reason'=
: 'vm not running on this host', 'health': 'bad', 'vm':<br>> 'down', 'de=
tail': 'unknown'}, 'score': 2400, 'maintenance': False,<br>> 'host-ts': =
1448}<br>> MainThread::INFO::2014-12-19<br>> 13:10:14,657::state_mach=
ine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh=
)<br>> Host 10.0.0.92 (id 3): {'extra':<br>> 'metadata_parse_version=
=3D1\nmetadata_feature_version=3D1\ntimestamp=3D987<br>> (Fri Dec 19 10:=
09:58<br>> 2014)\nhost-id=3D3\nscore=3D2400\nmaintenance=3DFalse\nstate=
=3DEngineDown\n',<br>> 'hostname': '10.0.0.92', 'alive': True, 'host-id'=
: 3, 'engine-status':<br>> {'reason': 'vm not running on this host', 'he=
alth': 'bad', 'vm':<br>> 'down', 'detail': 'unknown'}, 'score': 2400, 'm=
aintenance': False,<br>> 'host-ts': 987}<br>> MainThread::INFO::2014-=
12-19<br>> 13:10:14,658::state_machine::168::ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine::(refresh)<br>> Local (id 1): {'engine-heal=
th': {'health': 'good', 'vm': 'up',<br>> 'detail': 'up'}, 'bridge': True=
, 'mem-free': 1079.0, 'maintenance':<br>> False, 'cpu-load': 0.0269, 'ga=
teway': True}<br>> MainThread::INFO::2014-12-19<br>> 13:10:14,904::ho=
sted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::=
(start_monitoring)<br>> Current state EngineUp (score: 2400)<br>> Mai=
nThread::INFO::2014-12-19<br>> 13:10:14,904::hosted_engine::332::ovirt_h=
osted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>>=
; Best remote host 10.0.0.93 (id: 2, score: 2400)<br>> MainThread::INFO:=
:2014-12-19<br>> 13:10:25,210::hosted_engine::327::ovirt_hosted_engine_h=
a.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current stat=
e EngineUp (score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:10=
:25,210::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hos=
tedEngine::(start_monitoring)<br>> Best remote host 10.0.0.93 (id: 2, sc=
ore: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:10:35,499::hoste=
d_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(st=
art_monitoring)<br>> Current state EngineUp (score: 2400)<br>> MainTh=
read::INFO::2014-12-19<br>> 13:10:35,499::hosted_engine::332::ovirt_host=
ed_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> B=
est remote host 10.0.0.93 (id: 2, score: 2400)<br>> MainThread::INFO::20=
14-12-19<br>> 13:10:45,784::hosted_engine::327::ovirt_hosted_engine_ha.a=
gent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state E=
ngineUp (score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:10:45=
,785::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted=
Engine::(start_monitoring)<br>> Best remote host 10.0.0.93 (id: 2, score=
: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:10:56,070::hosted_e=
ngine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start=
_monitoring)<br>> Current state EngineUp (score: 2400)<br>> MainThrea=
d::INFO::2014-12-19<br>> 13:10:56,070::hosted_engine::332::ovirt_hosted_=
engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Best=
 remote host 10.0.0.93 (id: 2, score: 2400)<br>> MainThread::INFO::2014-=
12-19<br>> 13:11:06,109::states::394::ovirt_hosted_engine_ha.agent.hoste=
d_engine.HostedEngine::(consume)<br>> Engine vm running on localhost<br>=
> MainThread::INFO::2014-12-19<br>> 13:11:06,359::hosted_engine::327:=
:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring=
)<br>> Current state EngineUp (score: 2400)<br>> MainThread::INFO::20=
14-12-19<br>> 13:11:06,359::hosted_engine::332::ovirt_hosted_engine_ha.a=
gent.hosted_engine.HostedEngine::(start_monitoring)<br>> Best remote hos=
t 10.0.0.93 (id: 2, score: 2400)<br>> MainThread::INFO::2014-12-19<br>&g=
t; 13:11:16,658::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_en=
gine.HostedEngine::(start_monitoring)<br>> Current state EngineUp (score=
: 2400)<br>> MainThread::INFO::2014-12-19<br>> 13:11:16,658::hosted_e=
ngine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start=
_monitoring)<br>> Best remote host 10.0.0.93 (id: 2, score: 2400)<br>>=
; MainThread::INFO::2014-12-19<br>> 13:11:26,991::hosted_engine::327::ov=
irt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<b=
r>> Current state EngineUp (score: 2400)<br>> MainThread::INFO::2014-=
12-19<br>> 13:11:26,991::hosted_engine::332::ovirt_hosted_engine_ha.agen=
t.hosted_engine.HostedEngine::(start_monitoring)<br>> Best remote host 1=
0.0.0.93 (id: 2, score: 2400)<br>> MainThread::INFO::2014-12-19<br>> =
13:11:37,341::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engin=
e.HostedEngine::(start_monitoring)<br>> Current state EngineUp (score: 2=
400)<br>> MainThread::INFO::2014-12-19<br>> 13:11:37,341::hosted_engi=
ne::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_mo=
nitoring)<br>> Best remote host 10.0.0.93 (id: 2, score: 2400)<br>> -=
---<br>><br>> 10.0.0.93 (hosted-engine-2)<br>> MainThread::INFO::2=
014-12-19<br>> 10:12:18,339::hosted_engine::327::ovirt_hosted_engine_ha.=
agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state =
EngineDown (score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 10:12=
:18,339::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hos=
tedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id: 1, sc=
ore: 2400)<br>> MainThread::INFO::2014-12-19<br>> 10:12:28,651::hoste=
d_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(st=
art_monitoring)<br>> Current state EngineDown (score: 2400)<br>> Main=
Thread::INFO::2014-12-19<br>> 10:12:28,652::hosted_engine::332::ovirt_ho=
sted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>>=
 Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThread::INFO::=
2014-12-19<br>> 10:12:39,010::hosted_engine::327::ovirt_hosted_engine_ha=
.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current state=
 EngineDown (score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 10:1=
2:39,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Ho=
stedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id: 1, s=
core: 2400)<br>> MainThread::INFO::2014-12-19<br>> 10:12:49,338::host=
ed_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(s=
tart_monitoring)<br>> Current state EngineDown (score: 2400)<br>> Mai=
nThread::INFO::2014-12-19<br>> 10:12:49,338::hosted_engine::332::ovirt_h=
osted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>>=
; Best remote host 10.0.0.94 (id: 1, score: 2400)<br>> MainThread::INFO:=
:2014-12-19<br>> 10:12:59,642::hosted_engine::327::ovirt_hosted_engine_h=
a.agent.hosted_engine.HostedEngine::(start_monitoring)<br>> Current stat=
e EngineDown (score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 10:=
12:59,642::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.H=
ostedEngine::(start_monitoring)<br>> Best remote host 10.0.0.94 (id: 1, =
score: 2400)<br>> MainThread::INFO::2014-12-19<br>> 10:13:10,010::hos=
ted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(=
start_monitoring)<br>> Current state EngineDown (score: 2400)<br>> Ma=
inThread::INFO::2014-12-19<br>> 10:13:10,010::hosted_engine::332::ovirt_=
hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>&g=
t; Best remote host 10.0.0.94 (id: 1, score: 2400)<br>><br>><br>> =
10.0.0.92(hosted-engine-3)<br>> same as 10.0.0.93<br>> --<br>><br>=
> -----Original Message-----<br>> From: Simone Tiraboschi [mailto:sti=
rabos@redhat.com]<br>> Sent: Friday, December 19, 2014 12:28 AM<br>> =
To: Yue, Cong<br>> Cc: users@ovirt.org<mailto:users@ovirt.org><=
mailto:users@ovirt.org><br>> Subject: Re: [ovirt-users] VM failover w=
ith ovirt3.5<br>><br>><br>><br>> ----- Original Message -----<b=
r>> From: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@a=
lliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>><br>> T=
o: users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org&=
gt;<br>> Sent: Friday, December 19, 2014 2:14:33 AM<br>> Subject: [ov=
irt-users] VM failover with ovirt3.5<br>><br>><br>><br>> Hi<br>=
><br>><br>><br>> In my environment, I have 3 ovirt nodes as one=
 cluster. And on top of<br>> host-1, there is one vm to host ovirt engin=
e.<br>><br>> Also I have one external storage for the cluster to use =
as data domain<br>> of engine and data.<br>><br>> I confirmed live=
 migration works well in my environment.<br>><br>> But it seems very =
buggy for VM failover if I try to force to shut down<br>> one ovirt node=
. Sometimes the VM in the node which is shutdown can<br>> migrate to oth=
er host, but it take more than several minutes.<br>><br>> Sometimes, =
it can not migrate at all. Sometimes, only when the host is<br>> back, t=
he VM is beginning to move.<br>><br>> Can you please check or share t=
he logs under /var/log/ovirt-hosted-engine-ha/<br>> ?<br>><br>> Is=
 there some documentation to explain how VM failover is working? And<br>>=
; is there some bugs reported related with this?<br>><br>> http://www=
.ovirt.org/Features/Self_Hosted_Engine#Agent_State_Diagram<br>><br>> =
Thanks in advance,<br>><br>> Cong<br>><br>><br>><br>><br>=
> This e-mail message is for the sole use of the intended recipient(s)<b=
r>> and may contain confidential and privileged information. Any<br>>=
 unauthorized review, use, disclosure or distribution is prohibited. If<br>=
> you are not the intended recipient, please contact the sender by reply=
<br>> e-mail and destroy all copies of the original message. If you are =
the<br>> intended recipient, please be advised that the content of this =
message<br>> is subject to access, review and disclosure by the sender's=
 e-mail System<br>> Administrator.<br>><br>> _____________________=
__________________________<br>> Users mailing list<br>> Users@ovirt.o=
rg<mailto:Users@ovirt.org><mailto:Users@ovirt.org><br>> http=
://lists.ovirt.org/mailman/listinfo/users<br>><br>> This e-mail messa=
ge is for the sole use of the intended recipient(s) and may<br>> contain=
 confidential and privileged information. Any unauthorized review,<br>> =
use, disclosure or distribution is prohibited. If you are not the intended<=
br>> recipient, please contact the sender by reply e-mail and destroy al=
l copies<br>> of the original message. If you are the intended recipient=
, please be<br>> advised that the content of this message is subject to =
access, review and<br>> disclosure by the sender's e-mail System Adminis=
trator.<br>><br>><br>> This e-mail message is for the sole use of =
the intended recipient(s) and may contain confidential and privileged infor=
mation. Any unauthorized review, use, disclosure or distribution is prohibi=
ted. If you are not the intended recipient, please contact the sender by re=
ply e-mail and destroy all copies of the original message. If you are the i=
ntended recipient, please be advised that the content of this message is su=
bject to access, review and disclosure by the sender's e-mail System Admini=
strator.<br>> _______________________________________________<br>> Us=
ers mailing list<br>> Users@ovirt.org<mailto:Users@ovirt.org><m=
ailto:Users@ovirt.org><br>> http://lists.ovirt.org/mailman/listinfo/u=
sers<br>><br>> ________________________________<br>> This e-mail m=
essage is for the sole use of the intended recipient(s) and may contain con=
fidential and privileged information. Any unauthorized review, use, disclos=
ure or distribution is prohibited. If you are not the intended recipient, p=
lease contact the sender by reply e-mail and destroy all copies of the orig=
inal message. If you are the intended recipient, please be advised that the=
 content of this message is subject to access, review and disclosure by the=
 sender's e-mail System Administrator.<br>><br>> ____________________=
____________<br>> This e-mail message is for the sole use of the intende=
d recipient(s) and may contain confidential and privileged information. Any=
 unauthorized review, use, disclosure or distribution is prohibited. If you=
 are not the intended recipient, please contact the sender by reply e-mail =
and destroy all copies of the original message. If you are the intended rec=
ipient, please be advised that the content of this message is subject to ac=
cess, review and disclosure by the sender's e-mail System Administrator.<br=
thorized review, use, disclosure or distribution is prohibited. If you are =
not the intended recipient, please contact the sender by reply e-mail and d=
estroy all copies of the original message. If you are the intended recipien=
t, please be advised that the content of this message is subject to access,=
 review and disclosure by the sender's e-mail System Administrator.<br><div=
...
<br></div><br>------------------------------<br><div><br></div>___________=
____________________________________<br>Users mailing list<br>Users@ovirt.o=
rg<br>http://lists.ovirt.org/mailman/listinfo/users<br><div><br></div><br>E=
nd of Users Digest, Vol 39, Issue 169<br>**********************************=
****<br></div><div><br></div></div></body></html>
------=_Part_1875460_365779577.1419876418683--

Nikolai Sednev

tags

participants (1)