December 2014 - Users - Ovirt List Archives

Cannot start VM - pauses due to storage error

by Brent Hartzell

------=_NextPart_000_3233_01D021BB.C6C2CAC0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello, All of the sudden stared getting the errors below. Can't start VM's, they immediately go into paused state. Thread-51856::DEBUG::2014-12-27 09:56:08,812::vm::486::vm.Vm::(_getUserCpuTuneInfo) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not set Thread-51858::DEBUG::2014-12-27 09:56:11,953::vm::486::vm.Vm::(_getUserCpuTuneInfo) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not set Thread-51860::DEBUG::2014-12-27 09:56:12,856::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'VM.cont' in bridge with {u'vmID': u'e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d'} libvirtEventLoop::DEBUG::2014-12-27 09:56:13,016::vm::5461::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::event Resumed detail 0 opaque None libvirtEventLoop::DEBUG::2014-12-27 09:56:13,021::vm::5461::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::event Resumed detail 0 opaque None libvirtEventLoop::INFO::2014-12-27 09:56:13,026::vm::4780::vm.Vm::(_onIOError) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::abnormal vm stop device virtio-disk0 error eother libvirtEventLoop::DEBUG::2014-12-27 09:56:13,029::vm::5461::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::event Suspended detail 2 opaque None Thread-51863::DEBUG::2014-12-27 09:56:15,072::vm::486::vm.Vm::(_getUserCpuTuneInfo) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not set Thread-51869::DEBUG::2014-12-27 09:56:18,120::vm::486::vm.Vm::(_getUserCpuTuneInfo) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not set Thread-51872::DEBUG::2014-12-27 09:56:21,154::vm::486::vm.Vm::(_getUserCpuTuneInfo) vmId=`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not set ------=_NextPart_000_3233_01D021BB.C6C2CAC0 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><META = HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3Dus-ascii"><meta name=3DGenerator content=3D"Microsoft Word 15 = (filtered medium)"><style></style></head><body lang=3DEN-US = link=3D"#0563C1" vlink=3D"#954F72"><div class=3DWordSection1><p = class=3DMsoNormal>Hello,<o:p></o:p></p><p = class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>All of the = sudden stared getting the errors below. Can’t start VM’s, = they immediately go into paused state.<o:p></o:p></p><p = class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal><o:p> </o:p></p><p = class=3DMsoNormal>Thread-51856::DEBUG::2014-12-27 = 09:56:08,812::vm::486::vm.Vm::(_getUserCpuTuneInfo) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not = set<o:p></o:p></p><p class=3DMsoNormal>Thread-51858::DEBUG::2014-12-27 = 09:56:11,953::vm::486::vm.Vm::(_getUserCpuTuneInfo) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not = set<o:p></o:p></p><p class=3DMsoNormal>Thread-51860::DEBUG::2014-12-27 = 09:56:12,856::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) = Calling 'VM.cont' in bridge with {u'vmID': = u'e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d'}<o:p></o:p></p><p = class=3DMsoNormal>libvirtEventLoop::DEBUG::2014-12-27 = 09:56:13,016::vm::5461::vm.Vm::(_onLibvirtLifecycleEvent) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::event Resumed detail 0 = opaque None<o:p></o:p></p><p = class=3DMsoNormal>libvirtEventLoop::DEBUG::2014-12-27 = 09:56:13,021::vm::5461::vm.Vm::(_onLibvirtLifecycleEvent) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::event Resumed detail 0 = opaque None<o:p></o:p></p><p = class=3DMsoNormal>libvirtEventLoop::INFO::2014-12-27 = 09:56:13,026::vm::4780::vm.Vm::(_onIOError) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::abnormal vm stop device = virtio-disk0 error eother<o:p></o:p></p><p = class=3DMsoNormal>libvirtEventLoop::DEBUG::2014-12-27 = 09:56:13,029::vm::5461::vm.Vm::(_onLibvirtLifecycleEvent) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::event Suspended detail 2 = opaque None<o:p></o:p></p><p = class=3DMsoNormal>Thread-51863::DEBUG::2014-12-27 = 09:56:15,072::vm::486::vm.Vm::(_getUserCpuTuneInfo) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not = set<o:p></o:p></p><p class=3DMsoNormal>Thread-51869::DEBUG::2014-12-27 = 09:56:18,120::vm::486::vm.Vm::(_getUserCpuTuneInfo) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not = set<o:p></o:p></p><p class=3DMsoNormal>Thread-51872::DEBUG::2014-12-27 = 09:56:21,154::vm::486::vm.Vm::(_getUserCpuTuneInfo) = vmId=3D`e3d75e55-2b41-4f0b-8d2a-16f8fde2ba0d`::Domain Metadata is not = set<o:p></o:p></p></div></body></html> ------=_NextPart_000_3233_01D021BB.C6C2CAC0--

9 years, 11 months

2
1
0 / 0

1. Re: ??: bond mode balance-alb (Jorick Astrego)

by Nikolai Sednev

------=_Part_2052610_1445178490.1419935668210 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Mode 2 will do the job the best way for you in case of static LAG supported only at the switch's side, I'd advise using of xmit_hash_policy layer3+4, so you'll get better distribution for your DC. Thanks in advance. Best regards, Nikolai ____________________ Nikolai Sednev Senior Quality Engineer at Compute team Red Hat Israel 34 Jerusalem Road, Ra'anana, Israel 43501 Tel: +972 9 7692043 Mobile: +972 52 7342734 Email: nsednev(a)redhat.com IRC: nsednev ----- Original Message ----- From: users-request(a)ovirt.org To: users(a)ovirt.org Sent: Tuesday, December 30, 2014 2:12:58 AM Subject: Users Digest, Vol 39, Issue 173 Send Users mailing list submissions to users(a)ovirt.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.ovirt.org/mailman/listinfo/users or, via email, send a message with subject or body 'help' to users-request(a)ovirt.org You can reach the person managing the list at users-owner(a)ovirt.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Users digest..." Today's Topics: 1. Re: ??: bond mode balance-alb (Jorick Astrego) 2. Re: ??: bond mode balance-alb (Jorick Astrego) 3. HostedEngine Deployment Woes (Mikola Rose) ---------------------------------------------------------------------- Message: 1 Date: Mon, 29 Dec 2014 20:13:40 +0100 From: Jorick Astrego <j.astrego(a)netbulae.eu> To: users(a)ovirt.org Subject: Re: [ovirt-users] ??: bond mode balance-alb Message-ID: <54A1A7E4.90308(a)netbulae.eu> Content-Type: text/plain; charset="utf-8" On 12/29/2014 12:56 AM, Dan Kenigsberg wrote: > On Fri, Dec 26, 2014 at 12:39:45PM -0600, Blaster wrote: >> On 12/23/2014 2:55 AM, Dan Kenigsberg wrote: >>> Bug 1094842 - Bonding modes 0, 5 and 6 should be avoided for VM networks >>> https://bugzilla.redhat.com/show_bug.cgi?id=1094842#c0 >> Dan, >> >> What is bad about these modes that oVirt can't use them? > I can only quote jpirko's workds from the link above: > > Do not use tlb or alb in bridge, never! It does not work, that's it. The reason > is it mangles source macs in xmit frames and arps. When it is possible, just > use mode 4 (lacp). That should be always possible because all enterprise > switches support that. Generally, for 99% of use cases, you *should* use mode > 4. There is no reason to use other modes. > This switch is more of an office switch and only supports part of the 802.3ad standard: PowerConnect* *2824 Scalable from small workgroups to dense access solutions, the 2824 offers 24-port flexibility plus two combo small?form?factor pluggable (SFP) ports for connecting the switch to other networking equipment located beyond the 100 m distance limitations of copper cabling. Industry-standard link aggregation adhering to IEEE 802.3ad standards (static support only, LACP not supported) So the only way to have some kind of bonding without buying more expensive switches, is using balance-rr (mode=0), balance-xor (mode=2) or broadcast (modes=3). >> I just tested mode 4, and the LACP with Fedora 20 appears to not be >> compatible with the LAG mode on my Dell 2824. >> >> Would there be any issues with bringing two NICS into the VM and doing >> balance-alb at the guest level? >> Kind regards, Jorick Astrego Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts ---------------- Tel: 053 20 30 270 info(a)netbulae.eu Staalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ---------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ovirt.org/pipermail/users/attachments/20141229/dfacba22/atta...> ------------------------------ Message: 2 Date: Mon, 29 Dec 2014 20:14:55 +0100 From: Jorick Astrego <j.astrego(a)netbulae.eu> To: users(a)ovirt.org Subject: Re: [ovirt-users] ??: bond mode balance-alb Message-ID: <54A1A82F.1090100(a)netbulae.eu> Content-Type: text/plain; charset="utf-8" On 12/29/2014 12:56 AM, Dan Kenigsberg wrote: > On Fri, Dec 26, 2014 at 12:39:45PM -0600, Blaster wrote: >> On 12/23/2014 2:55 AM, Dan Kenigsberg wrote: >>> Bug 1094842 - Bonding modes 0, 5 and 6 should be avoided for VM networks >>> https://bugzilla.redhat.com/show_bug.cgi?id=1094842#c0 >> Sorry, no mode 0. So only mode 2 or 3 for your environment.... Kind regards, Jorick Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts ---------------- Tel: 053 20 30 270 info(a)netbulae.eu Staalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ---------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ovirt.org/pipermail/users/attachments/20141229/41da033b/atta...> ------------------------------ Message: 3 Date: Tue, 30 Dec 2014 00:12:52 +0000 From: Mikola Rose <mrose(a)power-soft.com> To: "users(a)ovirt.org" <users(a)ovirt.org> Subject: [ovirt-users] HostedEngine Deployment Woes Message-ID: <F992C848-E4EB-468E-83F4-37646EDB3E62(a)power-soft.com> Content-Type: text/plain; charset="us-ascii" Hi List Members; I have been struggling with deploying oVirt hosted engine I keep running into a timeout during the "Misc Configuration" any suggestion on how I can trouble shoot this? Redhat 2.6.32-504.3.3.el6.x86_64 Installed Packages ovirt-host-deploy.noarch 1.2.5-1.el6ev @rhel-6-server-rhevm-3.4-rpms ovirt-host-deploy-java.noarch 1.2.5-1.el6ev @rhel-6-server-rhevm-3.4-rpms ovirt-hosted-engine-ha.noarch 1.1.6-3.el6ev @rhel-6-server-rhevm-3.4-rpms ovirt-hosted-engine-setup.noarch 1.1.5-1.el6ev @rhel-6-server-rhevm-3.4-rpms rhevm-setup-plugin-ovirt-engine.noarch 3.4.4-2.2.el6ev @rhel-6-server-rhevm-3.4-rpms rhevm-setup-plugin-ovirt-engine-common.noarch 3.4.4-2.2.el6ev @rhel-6-server-rhevm-3.4-rpms Please confirm installation settings (Yes, No)[No]: Yes [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Configuring libvirt [ INFO ] Configuring VDSM [ INFO ] Starting vdsmd [ INFO ] Waiting for VDSM hardware info [ INFO ] Waiting for VDSM hardware info [ INFO ] Connecting Storage Domain [ INFO ] Connecting Storage Pool [ INFO ] Verifying sanlock lockspace initialization [ INFO ] sanlock lockspace already initialized [ INFO ] sanlock metadata already initialized [ INFO ] Creating VM Image [ INFO ] Disconnecting Storage Pool [ INFO ] Start monitoring domain [ ERROR ] Failed to execute stage 'Misc configuration': The read operation timed out [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination 2014-12-29 14:53:41 DEBUG otopi.plugins.ovirt_hosted_engine_setup.sanlock.lockspace lockspace._misc:133 Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /rhev/data-center/mnt/192.168.0.75:_Volumes_Raid1/8094d528-7aa2-4c28-839f-73d7c8bcfebb/ha_agent/hosted-engine.lockspace) 2014-12-29 14:53:41 INFO otopi.plugins.ovirt_hosted_engine_setup.sanlock.lockspace lockspace._misc:144 sanlock lockspace already initialized 2014-12-29 14:53:41 INFO otopi.plugins.ovirt_hosted_engine_setup.sanlock.lockspace lockspace._misc:157 sanlock metadata already initialized 2014-12-29 14:53:41 DEBUG otopi.context context._executeMethod:138 Stage misc METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.image.Plugin._misc 2014-12-29 14:53:41 INFO otopi.plugins.ovirt_hosted_engine_setup.vm.image image._misc:162 Creating VM Image 2014-12-29 14:53:41 DEBUG otopi.plugins.ovirt_hosted_engine_setup.vm.image image._misc:163 createVolume 2014-12-29 14:53:42 DEBUG otopi.plugins.ovirt_hosted_engine_setup.vm.image image._misc:184 Created volume d8e7eed4-c763-4b3d-8a71-35f2d692a73d, request was: - image: 9043e535-ea94-41f8-98df-6fdbfeb107c3 - volume: e6a9291d-ac21-4a95-b43c-0d6e552baaa2 2014-12-29 14:53:42 DEBUG otopi.ovirt_hosted_engine_setup.tasks tasks.wait:48 Waiting for existing tasks to complete 2014-12-29 14:53:43 DEBUG otopi.ovirt_hosted_engine_setup.tasks tasks.wait:48 Waiting for existing tasks to complete 2014-12-29 14:53:43 DEBUG otopi.context context._executeMethod:138 Stage misc METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk.Plugin._misc 2014-12-29 14:53:43 DEBUG otopi.context context._executeMethod:144 condition False 2014-12-29 14:53:43 DEBUG otopi.context context._executeMethod:138 Stage misc METHOD otopi.plugins.ovirt_hosted_engine_setup.storage.storage.Plugin._disconnect_pool 2014-12-29 14:53:43 INFO otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._disconnect_pool:971 Disconnecting Storage Pool 2014-12-29 14:53:43 DEBUG otopi.ovirt_hosted_engine_setup.tasks tasks.wait:48 Waiting for existing tasks to complete 2014-12-29 14:53:43 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._spmStop:602 spmStop 2014-12-29 14:53:43 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._spmStop:611 2014-12-29 14:53:43 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._storagePoolConnection:573 disconnectStoragePool 2014-12-29 14:53:45 INFO otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._disconnect_pool:975 Start monitoring domain 2014-12-29 14:53:45 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._startMonitoringDomain:529 _startMonitoringDomain 2014-12-29 14:53:46 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._startMonitoringDomain:534 {'status': {'message': 'OK', 'code': 0}} 2014-12-29 14:53:51 DEBUG otopi.ovirt_hosted_engine_setup.tasks tasks.wait:127 Waiting for domain monitor 2014-12-29 14:54:51 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 976, in _disconnect_pool self._startMonitoringDomain() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 539, in _startMonitoringDomain waiter.wait(self.environment[ohostedcons.StorageEnv.SD_UUID]) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/tasks.py", line 128, in wait response = serv.s.getVdsStats() File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib64/python2.6/xmlrpclib.py", line 1237, in request errcode, errmsg, headers = h.getreply() File "/usr/lib64/python2.6/httplib.py", line 1064, in getreply response = self._conn.getresponse() File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse response.begin() File "/usr/lib64/python2.6/httplib.py", line 391, in begin version, status, reason = self._read_status() File "/usr/lib64/python2.6/httplib.py", line 349, in _read_status line = self.fp.readline() File "/usr/lib64/python2.6/socket.py", line 433, in readline data = recv(1) File "/usr/lib64/python2.6/ssl.py", line 215, in recv return self.read(buflen) File "/usr/lib64/python2.6/ssl.py", line 136, in read return self._sslobj.read(len) SSLError: The read operation timed out var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20141229145137-g8d2or.log -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ovirt.org/pipermail/users/attachments/20141230/899f724c/atta...> ------------------------------ _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users End of Users Digest, Vol 39, Issue 173 ************************************** ------=_Part_2052610_1445178490.1419935668210 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: georgia,serif; font-size: 12pt; colo= r: #000000"><div><span style=3D"color: #333333; font-family: monospace; fon= t-size: 13.142857551574707px; background-color: #fdfdfd;" data-mce-style=3D= "color: #333333; font-family: monospace; font-size: 13.142857551574707px; b= ackground-color: #fdfdfd;">Mode 2 will do the job the best way for you in c= ase of static LAG supported only at the switch's side, I'd advise using of&= nbsp;xmit_hash_policy layer3+4, so you'll get better distribution for your = DC.</span></div><div><br></div><div><span name=3D"x"></span><br>Thanks in a= dvance.<br><div><br></div>Best regards,<br>Nikolai<br>____________________<= br>Nikolai Sednev<br>Senior Quality Engineer at Compute team<br>Red Hat Isr= ael<br>34 Jerusalem Road,<br>Ra'anana, Israel 43501<br><div><br></div>Tel: =       +972   9 7692043<br>Mobile: +972 52 7342734<br>Em= ail: nsednev(a)redhat.com<br>IRC: nsednev<span name=3D"x"></span><br></div><d= iv><br></div><hr id=3D"zwchr"><div style=3D"color:#000;font-weight:normal;f= ont-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-seri= f;font-size:12pt;" data-mce-style=3D"color: #000; font-weight: normal; font= -style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-se= rif; font-size: 12pt;"><b>From: </b>users-request(a)ovirt.org<br><b>To: </b>u= sers(a)ovirt.org<br><b>Sent: </b>Tuesday, December 30, 2014 2:12:58 AM<br><b>= Subject: </b>Users Digest, Vol 39, Issue 173<br><div><br></div>Send Users m= ailing list submissions to<br>       &nb= sp;users(a)ovirt.org<br><div><br></div>To subscribe or unsubscribe via the Wo= rld Wide Web, visit<br>        http= ://lists.ovirt.org/mailman/listinfo/users<br>or, via email, send a message = with subject or body 'help' to<br>      &nbsp= ; users-request(a)ovirt.org<br><div><br></div>You can reach the person m= anaging the list at<br>        user= s-owner(a)ovirt.org<br><div><br></div>When replying, please edit your Subject= line so it is more specific<br>than "Re: Contents of Users digest..."<br><= div><br></div><br>Today's Topics:<br><div><br></div>   1. Re: &nb= sp;??: bond mode balance-alb (Jorick Astrego)<br>   2. Re:  = ??: bond mode balance-alb (Jorick Astrego)<br>   3.  HostedE= ngine Deployment Woes (Mikola Rose)<br><div><br></div><br>-----------------= -----------------------------------------------------<br><div><br></div>Mes= sage: 1<br>Date: Mon, 29 Dec 2014 20:13:40 +0100<br>From: Jorick Astrego &l= t;j.astrego(a)netbulae.eu><br>To: users(a)ovirt.org<br>Subject: Re: [ovirt-u= sers] ??: bond mode balance-alb<br>Message-ID: <54A1A7E4.90308(a)netbulae.= eu><br>Content-Type: text/plain; charset=3D"utf-8"<br><div><br></div><br= >On 12/29/2014 12:56 AM, Dan Kenigsberg wrote:<br>> On Fri, Dec 26, 2014= at 12:39:45PM -0600, Blaster wrote:<br>>> On 12/23/2014 2:55 AM, Dan= Kenigsberg wrote:<br>>>> Bug 1094842 - Bonding modes 0, 5 and 6 s= hould be avoided for VM networks<br>>>> https://bugzilla.redhat.co= m/show_bug.cgi?id=3D1094842#c0<br>>> Dan,<br>>><br>>> Wha= t is bad about these modes that oVirt can't use them?<br>> I can only qu= ote jpirko's workds from the link above:<br>><br>>     Do n= ot use tlb or alb in bridge, never! It does not work, that's it. The reason= <br>>     is it mangles source macs in xmit frames and arps. W= hen it is possible, just<br>>     use mode 4 (lacp). That shou= ld be always possible because all enterprise<br>>     switches= support that. Generally, for 99% of use cases, you *should* use mode<br>&g= t;     4. There is no reason to use other modes.<br>><br>This = switch is more of an office switch and only supports part of the<br>802.3ad= standard:<br><div><br></div><br>           &= nbsp;PowerConnect* *2824<br><div><br></div>    Scalable from= small workgroups to dense access solutions, the 2824<br>   &nbsp= ;offers 24-port flexibility plus two combo small?form?factor<br> &nbsp= ;  pluggable (SFP) ports for connecting the switch to other networking= <br>    equipment located beyond the 100 m distance limitati= ons of copper<br>    cabling.<br><div><br></div>  =  Industry-standard link aggregation adhering to IEEE 802.3ad<br>&nbsp= ;   standards (static support only, LACP not supported)<br><div><= br></div><br>So the only way to have some kind of bonding without buying mo= re<br>expensive switches, is using balance-rr (mode=3D0), balance-xor (mode= =3D2)<br>or broadcast (modes=3D3).<br>>> I just tested mode 4, and th= e LACP with Fedora 20 appears to not be<br>>> compatible with the LAG= mode on my Dell 2824.<br>>><br>>> Would there be any issues wi= th bringing two NICS into the VM and doing<br>>> balance-alb at the g= uest level?<br>>><br>Kind regards,<br><div><br></div>Jorick Astrego<b= r><div><br></div><br><div><br></div>Met vriendelijke groet, With kind regar= ds,<br><div><br></div>Jorick Astrego<br><div><br></div>Netbulae Virtualizat= ion Experts <br><div><br></div>----------------<br><div><br></div> &nb= sp;      Tel: 053 20 30 270   &nbsp= ;     info(a)netbulae.eu     &nb= sp;   Staalsteden 4-3A       &= nbsp; KvK 08198180<br>        =  Fax: 053 20 30 271         ww= w.netbulae.eu         7547 TA Ensch= ede         BTW NL821234584B01<br><= div><br></div>----------------<br><div><br></div>-------------- next part -= -------------<br>An HTML attachment was scrubbed...<br>URL: <http://list= s.ovirt.org/pipermail/users/attachments/20141229/dfacba22/attachment-0001.h= tml><br><div><br></div>------------------------------<br><div><br></div>= Message: 2<br>Date: Mon, 29 Dec 2014 20:14:55 +0100<br>From: Jorick Astrego= <j.astrego(a)netbulae.eu><br>To: users(a)ovirt.org<br>Subject: Re: [ovir= t-users] ??: bond mode balance-alb<br>Message-ID: <54A1A82F.1090100@netb= ulae.eu><br>Content-Type: text/plain; charset=3D"utf-8"<br><div><br></di= v><br>On 12/29/2014 12:56 AM, Dan Kenigsberg wrote:<br>> On Fri, Dec 26,= 2014 at 12:39:45PM -0600, Blaster wrote:<br>>> On 12/23/2014 2:55 AM= , Dan Kenigsberg wrote:<br>>>> Bug 1094842 - Bonding modes 0, 5 an= d 6 should be avoided for VM networks<br>>>> https://bugzilla.redh= at.com/show_bug.cgi?id=3D1094842#c0<br>>><br>Sorry, no mode 0. So onl= y mode 2 or 3 for your environment....<br><div><br></div>Kind regards,<br><= div><br></div>Jorick<br><div><br></div><br><div><br></div>Met vriendelijke = groet, With kind regards,<br><div><br></div>Jorick Astrego<br><div><br></di= v>Netbulae Virtualization Experts <br><div><br></div>----------------<br><d= iv><br></div>        Tel: 053 20 30= 270         info(a)netbulae.eu &nbsp= ;       Staalsteden 4-3A   &nb= sp;     KvK 08198180<br>    &n= bsp;    Fax: 053 20 30 271     &nbs= p;   www.netbulae.eu       &nb= sp; 7547 TA Enschede         B= TW NL821234584B01<br><div><br></div>----------------<br><div><br></div>----= ---------- next part --------------<br>An HTML attachment was scrubbed...<b= r>URL: <http://lists.ovirt.org/pipermail/users/attachments/20141229/41da= 033b/attachment-0001.html><br><div><br></div>---------------------------= ---<br><div><br></div>Message: 3<br>Date: Tue, 30 Dec 2014 00:12:52 +0000<b= r>From: Mikola Rose <mrose(a)power-soft.com><br>To: "users(a)ovirt.org" &= lt;users(a)ovirt.org><br>Subject: [ovirt-users] HostedEngine Deployment Wo= es<br>Message-ID: <F992C848-E4EB-468E-83F4-37646EDB3E62(a)power-soft.com&g= t;<br>Content-Type: text/plain; charset=3D"us-ascii"<br><div><br></div><br>= Hi List Members;<br><div><br></div>I have been struggling with deploying oV= irt hosted engine  I keep running into a timeout during the "Misc Conf= iguration"  any suggestion on how I can trouble shoot this?<br><div><b= r></div>Redhat 2.6.32-504.3.3.el6.x86_64<br><div><br></div>Installed Packag= es<br>ovirt-host-deploy.noarch             &n= bsp;                     =                     &nbsp= ;                     &nb= sp;             1.2.5-1.el6ev     &= nbsp;                    =                     &nbs= p;                     &n= bsp;   @rhel-6-server-rhevm-3.4-rpms<br>ovirt-host-deploy-java.noarch =                     &nbsp= ;                     &nb= sp;                     &= nbsp;                    = 1.2.5-1.el6ev                 &nbsp= ;                     &nb= sp;                     &= nbsp;           @rhel-6-server-rhevm-3.4-rpms<br>o= virt-hosted-engine-ha.noarch             &nbs= p;                     &n= bsp;                     =                     &nbsp= ;        1.1.6-3.el6ev         &nbs= p;                     &n= bsp;                     =                     @rhel= -6-server-rhevm-3.4-rpms<br>ovirt-hosted-engine-setup.noarch     =                     &nbsp= ;                     &nb= sp;                     &= nbsp;             1.1.5-1.el6ev    =                     &nbs= p;                     &n= bsp;                     =     @rhel-6-server-rhevm-3.4-rpms<br>rhevm-setup-plugin-ovirt-eng= ine.noarch                   &= nbsp;                    =                     &nbs= p;               3.4.4-2.2.el6ev   =                     &nbsp= ;                     &nb= sp;                     &= nbsp;   @rhel-6-server-rhevm-3.4-rpms<br>rhevm-setup-plugin-ovirt-engi= ne-common.noarch                 &n= bsp;                     =                     &nbsp= ;          3.4.4-2.2.el6ev       &n= bsp;                     =                     &nbsp= ;                     @rh= el-6-server-rhevm-3.4-rpms<br><div><br></div><br>      =    Please confirm installation settings (Yes, No)[No]: Yes<br>[ = INFO  ] Stage: Transaction setup<br>[ INFO  ] Stage: Misc configu= ration<br>[ INFO  ] Stage: Package installation<br>[ INFO  ] Stag= e: Misc configuration<br>[ INFO  ] Configuring libvirt<br>[ INFO &nbsp= ;] Configuring VDSM<br>[ INFO  ] Starting vdsmd<br>[ INFO  ] Wait= ing for VDSM hardware info<br>[ INFO  ] Waiting for VDSM hardware info= <br>[ INFO  ] Connecting Storage Domain<br>[ INFO  ] Connecting S= torage Pool<br>[ INFO  ] Verifying sanlock lockspace initialization<br= >[ INFO  ] sanlock lockspace already initialized<br>[ INFO  ] san= lock metadata already initialized<br>[ INFO  ] Creating VM Image<br>[ = INFO  ] Disconnecting Storage Pool<br>[ INFO  ] Start monitoring = domain<br>[ ERROR ] Failed to execute stage 'Misc configuration': The read = operation timed out<br>[ INFO  ] Stage: Clean up<br>[ INFO  ] Gen= erating answer file '/etc/ovirt-hosted-engine/answers.conf'<br>[ INFO &nbsp= ;] Stage: Pre-termination<br>[ INFO  ] Stage: Termination<br><div><br>= </div><br><div><br></div>2014-12-29 14:53:41 DEBUG otopi.plugins.ovirt_host= ed_engine_setup.sanlock.lockspace lockspace._misc:133 Ensuring lease for lo= ckspace hosted-engine, host id 1 is acquired (file: /rhev/data-center/mnt/1= 92.168.0.75:_Volumes_Raid1/8094d528-7aa2-4c28-839f-73d7c8bcfebb/ha_agent/ho= sted-engine.lockspace)<br>2014-12-29 14:53:41 INFO otopi.plugins.ovirt_host= ed_engine_setup.sanlock.lockspace lockspace._misc:144 sanlock lockspace alr= eady initialized<br>2014-12-29 14:53:41 INFO otopi.plugins.ovirt_hosted_eng= ine_setup.sanlock.lockspace lockspace._misc:157 sanlock metadata already in= itialized<br>2014-12-29 14:53:41 DEBUG otopi.context context._executeMethod= :138 Stage misc METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.image.Plu= gin._misc<br>2014-12-29 14:53:41 INFO otopi.plugins.ovirt_hosted_engine_set= up.vm.image image._misc:162 Creating VM Image<br>2014-12-29 14:53:41 DEBUG = otopi.plugins.ovirt_hosted_engine_setup.vm.image image._misc:163 createVolu= me<br>2014-12-29 14:53:42 DEBUG otopi.plugins.ovirt_hosted_engine_setup.vm.= image image._misc:184 Created volume d8e7eed4-c763-4b3d-8a71-35f2d692a73d, = request was:<br>- image: 9043e535-ea94-41f8-98df-6fdbfeb107c3<br>- volume: = e6a9291d-ac21-4a95-b43c-0d6e552baaa2<br>2014-12-29 14:53:42 DEBUG otopi.ovi= rt_hosted_engine_setup.tasks tasks.wait:48 Waiting for existing tasks to co= mplete<br>2014-12-29 14:53:43 DEBUG otopi.ovirt_hosted_engine_setup.tasks t= asks.wait:48 Waiting for existing tasks to complete<br>2014-12-29 14:53:43 = DEBUG otopi.context context._executeMethod:138 Stage misc METHOD otopi.plug= ins.ovirt_hosted_engine_setup.vm.boot_disk.Plugin._misc<br>2014-12-29 14:53= :43 DEBUG otopi.context context._executeMethod:144 condition False<br>2014-= 12-29 14:53:43 DEBUG otopi.context context._executeMethod:138 Stage misc ME= THOD otopi.plugins.ovirt_hosted_engine_setup.storage.storage.Plugin._discon= nect_pool<br>2014-12-29 14:53:43 INFO otopi.plugins.ovirt_hosted_engine_set= up.storage.storage storage._disconnect_pool:971 Disconnecting Storage Pool<= br>2014-12-29 14:53:43 DEBUG otopi.ovirt_hosted_engine_setup.tasks tasks.wa= it:48 Waiting for existing tasks to complete<br>2014-12-29 14:53:43 DEBUG o= topi.plugins.ovirt_hosted_engine_setup.storage.storage storage._spmStop:602= spmStop<br>2014-12-29 14:53:43 DEBUG otopi.plugins.ovirt_hosted_engine_set= up.storage.storage storage._spmStop:611<br>2014-12-29 14:53:43 DEBUG otopi.= plugins.ovirt_hosted_engine_setup.storage.storage storage._storagePoolConne= ction:573 disconnectStoragePool<br>2014-12-29 14:53:45 INFO otopi.plugins.o= virt_hosted_engine_setup.storage.storage storage._disconnect_pool:975 Start= monitoring domain<br>2014-12-29 14:53:45 DEBUG otopi.plugins.ovirt_hosted_= engine_setup.storage.storage storage._startMonitoringDomain:529 _startMonit= oringDomain<br>2014-12-29 14:53:46 DEBUG otopi.plugins.ovirt_hosted_engine_= setup.storage.storage storage._startMonitoringDomain:534 {'status': {'messa= ge': 'OK', 'code': 0}}<br>2014-12-29 14:53:51 DEBUG otopi.ovirt_hosted_engi= ne_setup.tasks tasks.wait:127 Waiting for domain monitor<br>2014-12-29 14:5= 4:51 DEBUG otopi.context context._executeMethod:152 method exception<br>Tra= ceback (most recent call last):<br>  File "/usr/lib/python2.6/sit= e-packages/otopi/context.py", line 142, in _executeMethod<br>   &= nbsp;method['method']()<br>  File "/usr/share/ovirt-hosted-engine= -setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", li= ne 976, in _disconnect_pool<br>    self._startMonitoringDoma= in()<br>  File "/usr/share/ovirt-hosted-engine-setup/scripts/../p= lugins/ovirt-hosted-engine-setup/storage/storage.py", line 539, in _startMo= nitoringDomain<br>    waiter.wait(self.environment[ohostedco= ns.StorageEnv.SD_UUID])<br>  File "/usr/lib/python2.6/site-packag= es/ovirt_hosted_engine_setup/tasks.py", line 128, in wait<br>   &= nbsp;response =3D serv.s.getVdsStats()<br>  File "/usr/lib64/pyth= on2.6/xmlrpclib.py", line 1199, in __call__<br>    return se= lf.__send(self.__name, args)<br>  File "/usr/lib64/python2.6/xmlr= pclib.py", line 1489, in __request<br>    verbose=3Dself.__v= erbose<br>  File "/usr/lib64/python2.6/xmlrpclib.py", line 1237, = in request<br>    errcode, errmsg, headers =3D h.getreply()<= br>  File "/usr/lib64/python2.6/httplib.py", line 1064, in getrep= ly<br>    response =3D self._conn.getresponse()<br> &nb= sp;File "/usr/lib64/python2.6/httplib.py", line 990, in getresponse<br>&nbs= p;   response.begin()<br>  File "/usr/lib64/python2.6/h= ttplib.py", line 391, in begin<br>    version, status, reaso= n =3D self._read_status()<br>  File "/usr/lib64/python2.6/httplib= .py", line 349, in _read_status<br>    line =3D self.fp.read= line()<br>  File "/usr/lib64/python2.6/socket.py", line 433, in r= eadline<br>    data =3D recv(1)<br>  File "/usr/li= b64/python2.6/ssl.py", line 215, in recv<br>    return self.= read(buflen)<br>  File "/usr/lib64/python2.6/ssl.py", line 136, i= n read<br>    return self._sslobj.read(len)<br>SSLError: The= read operation timed out<br><div><br></div><br><div><br></div><br>var/log/= ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20141229145137-g8d2or.l= og<br>-------------- next part --------------<br>An HTML attachment was scr= ubbed...<br>URL: <http://lists.ovirt.org/pipermail/users/attachments/201= 41230/899f724c/attachment.html><br><div><br></div>----------------------= --------<br><div><br></div>_______________________________________________<= br>Users mailing list<br>Users(a)ovirt.org<br>http://lists.ovirt.org/mailman/= listinfo/users<br><div><br></div><br>End of Users Digest, Vol 39, Issue 173= <br>**************************************<br></div><div><br></div></div></= body></html> ------=_Part_2052610_1445178490.1419935668210--

9 years, 11 months

3
2
0 / 0

engine-iso-uploader unexpected behaviour

by Steve Atkinson

When attempting use the engine-iso-uploader to drop ISOs in my iso storage domain I get the following results. Using engine-iso-uploader --iso-domain=[domain] upload [iso] does not work because the engine does not have access to our storage network. So it attempts to mount to an address that is not routable. I thought to resolve this by adding an interfaces to the Hosted Engine, only to find that I cannot modify the Engine's VM config from the GUI. I receive the error: Cannot add Interface. This VM is not managed by the engine. Actually, I get that error whenever I attempt to modify anything about the engine. Maybe this is expected behavior? I can't find any bestpractices regarding Hosted Engine administration. Alternatively, using engine-iso-uploader --nfs-server=[path] upload [iso] --verbose returns the following error: ERROR: local variable 'domain_type' referenced before assignment INFO: Use the -h option to see usage. DEBUG: Configuration: DEBUG: command: upload DEBUG: Traceback (most recent call last): DEBUG: File "/usr/bin/engine-iso-uploader", line 1440, in <module> DEBUG: isoup = ISOUploader(conf) DEBUG: File "/usr/bin/engine-iso-uploader", line 455, in __init__ DEBUG: self.upload_to_storage_domain() DEBUG: File "/usr/bin/engine-iso-uploader", line 1089, in upload_to_storage_domain DEBUG: elif domain_type in ('localfs', ): DEBUG: UnboundLocalError: local variable 'domain_type' referenced before assignment Engine is Self Hosted and is Version 3.5.0.1-1.el6. Thanks! -Steve

9 years, 11 months

2
1
0 / 0

feedback-on-oVirt-engine-3.5.0.1-1.el6

by bingozhou2013

------=_001_NextPart566401241025_=---- Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: base64 RGVhciBTaXIsDQoNCldoZW4gIEkgIHRyeSB0byBpbnN0YWxsIHRoZSBvVmlydC1lbmdpbmUgMy41 ICBpbiB0aGUgQ2VudE9TIDYuNiAgLkJlbG93IGVycm9yIGlzIHNob3cgOg0KDQotLT4gRmluaXNo ZWQgRGVwZW5kZW5jeSBSZXNvbHV0aW9uDQpFcnJvcjogUGFja2FnZTogb3ZpcnQtZW5naW5lLWJh Y2tlbmQtMy41LjAuMS0xLmVsNi5ub2FyY2ggKG92aXJ0LTMuNSkNCiAgICAgICAgICAgUmVxdWly ZXM6IG5vdm5jDQogWW91IGNvdWxkIHRyeSB1c2luZyAtLXNraXAtYnJva2VuIHRvIHdvcmsgYXJv dW5kIHRoZSBwcm9ibGVtDQogWW91IGNvdWxkIHRyeSBydW5uaW5nOiBycG0gLVZhIC0tbm9maWxl cyAtLW5vZGlnZXN0DQoNCg0KSSBoYXZlICBhZGRlZCB0aGUgIEVQRUwgc291cmNlIGFuZCBpbnN0 YWxsZWQgIG92aXJ0LXJlbGVhc2UzNS5ycG0gLGJ1dCBpdCBzdGlsbCBzaG93cyAicmVxdWlyZXM6 bm92bmMiIC4gUGxlYXNlIGhlbHAgbWUgdG8gY2hlY2sgdGhpcyAuVGhhbmsgeW91IHZlcnkgbXVj aCAhDQoNCg0KDQoNCmJpbmdvemhvdTIwMTM= ------=_001_NextPart566401241025_=---- Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML xmlns:o =3D "urn:schemas-microsoft-com:office:office"><HEAD> <META content=3D"text/html; charset=3Dus-ascii" http-equiv=3DContent-Type> <STYLE> BLOCKQUOTE { MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px; MARGIN-LEFT: 2em } OL { MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px } UL { MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px } P { MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px } BODY { LINE-HEIGHT: 1.5; FONT-FAMILY: Tahoma; COLOR: #000000; FONT-SIZE: 10.5pt } </STYLE> <META name=3DGENERATOR content=3D"MSHTML 9.00.8112.16599"></HEAD> <BODY style=3D"MARGIN: 10px"> <DIV>Dear Sir,</DIV> <DIV> </DIV> <DIV>When  I  try to install the oVirt-engine 3.5  in the C= entOS=20 6.6  .Below error is show :</DIV> <DIV> </DIV> <DIV> <DIV>--> Finished Dependency Resolution</DIV> <DIV style=3D""><SPAN=20 style=3D"BACKGROUND-COLOR: #ff0000">Error: Package: ovirt-engine= -backend-3.5.0.1-1.el6.noarch (ovirt-3.5)</SPAN></DIV> <DIV style=3D""><SPAN=20 style=3D"BACKGROUND-COLOR: #ff0000">      &n= bsp;    Requires: novnc</SPAN></DIV> <DIV> You could try using --skip-broken to&n= bsp;work around the problem</DIV> <DIV> You could try running: rpm -Va --= nofiles --nodigest</DIV></DIV> <DIV> </DIV> <DIV> </DIV> <DIV>I have  added the  EPEL source and installed =20 ovirt-release35.rpm ,but it still shows "requires:novnc" . Please help me = to=20 check this .Thank you very much !</DIV> <DIV> </DIV> <HR style=3D"WIDTH: 210px; HEIGHT: 1px" align=3Dleft color=3D#b5c4df SIZE= =3D1> <DIV><SPAN>bingozhou2013</SPAN></DIV></BODY></HTML> ------=_001_NextPart566401241025_=------

9 years, 11 months

3
2
0 / 0

domain storage of the type Glusterfs

by suporte＠logicworks.pt

------=_Part_25142_1582656685.1419956865500 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Concerning to Version 3.5.0.1-1.el6, For creating a domain storage of the type Glusterfs, do I have to create a host first? Thanks Jose -- Jose Ferradeira http://www.logicworks.pt ------=_Part_25142_1582656685.1419956865500 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html><body><div style="font-family: Times New Roman; font-size: 10pt; color: #000000"><div>Hi,<br></div><div><br></div><div><span class="version-text">Concerning to Version 3.5.0.1-1.el6,</span></div><div><span class="version-text">For creating a domain storage of the type Glusterfs, do I have to create a host first?<br></span></div><div><br class="version-text"></div><div>Thanks<br></div><div><br></div><div>Jose<br></div><div>-- <br></div><div><span name="x"></span><hr style="width: 100%; height: 2px;" data-mce-style="width: 100%; height: 2px;">Jose Ferradeira<br>http://www.logicworks.pt<br><span name="x"></span><br></div></div></body></html> ------=_Part_25142_1582656685.1419956865500--

9 years, 11 months

1
0
0 / 0

Centos 6.6 ovirt install fail. dependency novnc error

by Anders Hellquist

Hi ! I am trying to install ovirt 3.5 on a Centos 6.6 host Have tried both http://resources.ovirt.org/pub/yum-repo/ovirt-release35.rpm http://resources.ovirt.org/pub/yum-repo/ovirt-release35-snapshot.rpm While installing ovirt-engine, ovirt-engine-backend requires novnc and fails. --> Running transaction check ---> Package bind-libs.x86_64 32:9.8.2-0.30.rc1.el6_6.1 will be installed ---> Package ovirt-engine-backend.noarch 0:3.5.1-0.0.master.20141219191829.gite2b1d3a.el6 will be installed --> Processing Dependency: novnc for package: ovirt-engine-backend-3.5.1-0.0.master.20141219191829.gite2b1d3a.el6.noarch ---> Package perl-Pod-Escapes.x86_64 1:1.04-136.el6_6.1 will be installed ---> Package python-crypto.x86_64 0:2.0.1-22.el6 will be installed ---> Package setools-libs.x86_64 0:3.3.7-4.el6 will be installed --> Finished Dependency Resolution Error: Package: ovirt-engine-backend-3.5.1-0.0.master.20141219191829.gite2b1d3a.el6.noarch (ovirt-3.5-snapshot) Requires: novnc You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest Have looked for novnc package but maybe it is named something else.. Anyone ? Best regards, Anders

9 years, 11 months

5
6
0 / 0

Unable to reinstall hosts after network removal.

by Arman Khalatyan

Hello, I have a little trouble with ovirt 3.5 on CentOS6.6: I was removing all networks from all hosts. Then after removing network from data center the hosts went to unusable. Every time after reinstall the host claims that the network is not configured, but it s already removed from network tab in DC. Where from it gets the old configuration? the old interfaces also restored every time on the reinstalled hosts. Which DB table is in charge of dc-networks? Thanks, Arman. *********************************************************** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany ***********************************************************

9 years, 11 months

3
6
0 / 0

VM failover with ovirt3.5

by Yue, Cong

--_000_ED08B56256B38842A463A2A0804C5AC0326ACA4745svrcaexch1atg_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi In my environment, I have 3 ovirt nodes as one cluster. And on top of host-= 1, there is one vm to host ovirt engine. Also I have one external storage for the cluster to use as data domain of e= ngine and data. I confirmed live migration works well in my environment. But it seems very buggy for VM failover if I try to force to shut down one = ovirt node. Sometimes the VM in the node which is shutdown can migrate to o= ther host, but it take more than several minutes. Sometimes, it can not migrate at all. Sometimes, only when the host is back= , the VM is beginning to move. Is there some documentation to explain how VM failover is working? And is t= here some bugs reported related with this? Thanks in advance, Cong ________________________________ This e-mail message is for the sole use of the intended recipient(s) and ma= y contain confidential and privileged information. Any unauthorized review,= use, disclosure or distribution is prohibited. If you are not the intended= recipient, please contact the sender by reply e-mail and destroy all copie= s of the original message. If you are the intended recipient, please be adv= ised that the content of this message is subject to access, review and disc= losure by the sender's e-mail System Administrator. --_000_ED08B56256B38842A463A2A0804C5AC0326ACA4745svrcaexch1atg_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">Hi<o:p></o:p= ></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA"><o:p> <= /o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">In my enviro= nment, I have 3 ovirt nodes as one cluster. And on top of host-1, there is = one vm to host ovirt engine.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">Also I have = one external storage for the cluster to use as data domain of engine and da= ta.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">I confirmed = live migration works well in my environment.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">But it seems= very buggy for VM failover if I try to force to shut down one ovirt node. = Sometimes the VM in the node which is shutdown can migrate to other host, b= ut it take more than several minutes.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">Sometimes, i= t can not migrate at all. Sometimes, only when the host is back, the VM is = beginning to move.<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA"><o:p> <= /o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">Is there som= e documentation to explain how VM failover is working? And is there some bu= gs reported related with this?<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA"><o:p> <= /o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">Thanks in ad= vance,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA">Cong<o:p></o= :p></span></p> <p class=3D"MsoNormal"><span style=3D"mso-fareast-language:JA"><o:p> <= /o:p></span></p> </div> <br> <hr> <font face=3D"Arial" color=3D"Gray" size=3D"1">This e-mail message is for t= he sole use of the intended recipient(s) and may contain confidential and p= rivileged information. Any unauthorized review, use, disclosure or distribu= tion is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy = all copies of the original message. If you are the intended recipient, plea= se be advised that the content of this message is subject to access, review= and disclosure by the sender's e-mail System Administrator.<br> </font> </body> </html> --_000_ED08B56256B38842A463A2A0804C5AC0326ACA4745svrcaexch1atg_--

9 years, 11 months

3
16
0 / 0

bond mode balance-alb

by Blaster

I’m looking to setup bonding. I’m using Dell 2824 switches which support LAG (static 802.3ad apparently?) but not LACP (dynamic 802.3ad). Fedora only seems to support Dynamic mode from what I can tell. The next best choice seems to be balance-alb, but for some reason it’s not available as a mode in the drop down. Any reason why it’s missing? If I select custom and enter balance-alb, is that supported?

9 years, 12 months

5
6
0 / 0

Re: [ovirt-users] Users Digest, Vol 39, Issue 171

by Nikolai Sednev

------=_Part_1882534_1428653136.1419879250032 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Can you please provide engine.log from /var/log/ovirt-engine/engine.log and to try as follows: 1. Revert all tree hosts to maintenance-mode=none. 2. Check that engine up and running. 3. Turn one of the hosts that is not running the engine, to maintenance mode local. 4. Turn host that is running the engine to maintenance mode local. 5. Check that engine migrated to one and the only remaining host, that had not been put in to maintenance mode at all. Can you also provide your engine version, is it 3.4 something? Thanks in advance. Best regards, Nikolai ____________________ Nikolai Sednev Senior Quality Engineer at Compute team Red Hat Israel 34 Jerusalem Road, Ra'anana, Israel 43501 Tel: +972 9 7692043 Mobile: +972 52 7342734 Email: nsednev(a)redhat.com IRC: nsednev ----- Original Message ----- From: users-request(a)ovirt.org To: users(a)ovirt.org Sent: Monday, December 29, 2014 8:29:36 PM Subject: Users Digest, Vol 39, Issue 171 Send Users mailing list submissions to users(a)ovirt.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.ovirt.org/mailman/listinfo/users or, via email, send a message with subject or body 'help' to users-request(a)ovirt.org You can reach the person managing the list at users-owner(a)ovirt.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Users digest..." Today's Topics: 1. Re: VM failover with ovirt3.5 (Yue, Cong) ---------------------------------------------------------------------- Message: 1 Date: Mon, 29 Dec 2014 10:29:04 -0800 From: "Yue, Cong" <Cong_Yue(a)alliedtelesis.com> To: Artyom Lukianov <alukiano(a)redhat.com> Cc: "users(a)ovirt.org" <users(a)ovirt.org> Subject: Re: [ovirt-users] VM failover with ovirt3.5 Message-ID: <21D302CF-AD6F-4E8C-A373-52ADAC1C129B(a)alliedtelesis.com> Content-Type: text/plain; charset="utf-8" I disabled local maintenance mode for all hosts, and then only set the host where HE VM is there to local maintenance mode. The logs are as follows. During the migration of HE VM , it shows some fatal error happen. By the way, also HE VM can not work with live migration. Instead, other VMs can do live migration. --- [root@compute2-3 ~]# hosted-engine --set-maintenance --mode=local You have new mail in /var/spool/mail/root [root@compute2-3 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log MainThread::INFO::2014-12-29 13:16:12,435::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.92 (id: 3, score: 2400) MainThread::INFO::2014-12-29 13:16:22,711::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-29 13:16:22,711::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.92 (id: 3, score: 2400) MainThread::INFO::2014-12-29 13:16:32,978::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-29 13:16:32,978::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-29 13:16:43,272::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-29 13:16:43,272::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-29 13:16:53,316::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost MainThread::INFO::2014-12-29 13:16:53,562::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-29 13:16:53,562::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-29 13:17:03,600::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-29 13:17:03,611::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1419877023.61 type=state_transition detail=EngineUp-LocalMaintenanceMigrateVm hostname='compute2-3' MainThread::INFO::2014-12-29 13:17:03,672::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineUp-LocalMaintenanceMigrateVm) sent? sent MainThread::INFO::2014-12-29 13:17:03,911::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is 0 due to local maintenance mode MainThread::INFO::2014-12-29 13:17:03,912::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenanceMigrateVm (score: 0) MainThread::INFO::2014-12-29 13:17:03,912::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-29 13:17:03,960::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1419877023.96 type=state_transition detail=LocalMaintenanceMigrateVm-EngineMigratingAway hostname='compute2-3' MainThread::INFO::2014-12-29 13:17:03,980::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (LocalMaintenanceMigrateVm-EngineMigratingAway) sent? sent MainThread::INFO::2014-12-29 13:17:04,218::states::66::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_penalize_memory) Penalizing score by 400 due to low free memory MainThread::INFO::2014-12-29 13:17:04,218::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineMigratingAway (score: 2000) MainThread::INFO::2014-12-29 13:17:04,219::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::ERROR::2014-12-29 13:17:14,251::hosted_engine::867::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_monitor_migration) Failed to migrate Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 863, in _monitor_migration vm_id, File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/vds_client.py", line 85, in run_vds_client_cmd response['status']['message']) DetailedError: Error 12 from migrateStatus: Fatal error during migration MainThread::INFO::2014-12-29 13:17:14,262::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1419877034.26 type=state_transition detail=EngineMigratingAway-ReinitializeFSM hostname='compute2-3' MainThread::INFO::2014-12-29 13:17:14,263::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineMigratingAway-ReinitializeFSM) sent? ignored MainThread::INFO::2014-12-29 13:17:14,496::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state ReinitializeFSM (score: 0) MainThread::INFO::2014-12-29 13:17:14,496::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-29 13:17:24,536::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-29 13:17:24,547::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1419877044.55 type=state_transition detail=ReinitializeFSM-LocalMaintenance hostname='compute2-3' MainThread::INFO::2014-12-29 13:17:24,574::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (ReinitializeFSM-LocalMaintenance) sent? sent MainThread::INFO::2014-12-29 13:17:24,812::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-29 13:17:24,812::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-29 13:17:34,851::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-29 13:17:35,095::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-29 13:17:35,095::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-29 13:17:45,130::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-29 13:17:45,368::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-29 13:17:45,368::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) ^C [root@compute2-3 ~]# [root@compute2-3 ~]# hosted-engine --vm-status --== Host 1 status ==-- Status up-to-date : True Hostname : 10.0.0.94 Host ID : 1 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 0 Local maintenance : True Host timestamp : 1014956<tel:1014956> Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1014956<tel:1014956> (Mon Dec 29 13:20:19 2014) host-id=1 score=0 maintenance=True state=LocalMaintenance --== Host 2 status ==-- Status up-to-date : True Hostname : 10.0.0.93 Host ID : 2 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 2400 Local maintenance : False Host timestamp : 866019 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=866019 (Mon Dec 29 10:19:45 2014) host-id=2 score=2400 maintenance=False state=EngineDown --== Host 3 status ==-- Status up-to-date : True Hostname : 10.0.0.92 Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 2400 Local maintenance : False Host timestamp : 860493 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=860493 (Mon Dec 29 10:20:35 2014) host-id=3 score=2400 maintenance=False state=EngineDown [root@compute2-3 ~]# --- Thanks, Cong On 2014/12/29, at 8:43, "Artyom Lukianov" <alukiano(a)redhat.com<mailto:alukiano@redhat.com>> wrote: I see that HE vm run on host with ip 10.0.0.94, and two another hosts in "Local Maintenance" state, so vm will not migrate to any of them, can you try disable local maintenance on all hosts in HE environment and after enable "local maintenance" on host where HE vm run, and provide also output of hosted-engine --vm-status. Failover works in next way: 1) if host where run HE vm have score less by 800 that some other host in HE environment, HE vm will migrate on host with best score 2) if something happen to vm(kernel panic, crash of service...), agent will restart HE vm on another host in HE environment with positive score 3) if put to local maintenance host with HE vm, vm will migrate to another host with positive score Thanks. ----- Original Message ----- From: "Cong Yue" <Cong_Yue(a)alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com>> To: "Artyom Lukianov" <alukiano(a)redhat.com<mailto:alukiano@redhat.com>> Cc: "Simone Tiraboschi" <stirabos(a)redhat.com<mailto:stirabos@redhat.com>>, users(a)ovirt.org<mailto:users@ovirt.org> Sent: Monday, December 29, 2014 6:30:42 PM Subject: Re: [ovirt-users] VM failover with ovirt3.5 Thanks and the --vm-status log is as follows: [root@compute2-2 ~]# hosted-engine --vm-status --== Host 1 status ==-- Status up-to-date : True Hostname : 10.0.0.94 Host ID : 1 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 2400 Local maintenance : False Host timestamp : 1008087 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1008087<tel:1008087> (Mon Dec 29 11:25:51 2014) host-id=1 score=2400 maintenance=False state=EngineUp --== Host 2 status ==-- Status up-to-date : True Hostname : 10.0.0.93 Host ID : 2 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 Local maintenance : True Host timestamp : 859142 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=859142 (Mon Dec 29 08:25:08 2014) host-id=2 score=0 maintenance=True state=LocalMaintenance --== Host 3 status ==-- Status up-to-date : True Hostname : 10.0.0.92 Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 Local maintenance : True Host timestamp : 853615 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=853615 (Mon Dec 29 08:25:57 2014) host-id=3 score=0 maintenance=True state=LocalMaintenance You have new mail in /var/spool/mail/root [root@compute2-2 ~]# Could you please explain how VM failover works inside ovirt? Is there any other debug option I can enable to check the problem? Thanks, Cong On 2014/12/29, at 1:39, "Artyom Lukianov" <alukiano(a)redhat.com<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>> wrote: Can you also provide output of hosted-engine --vm-status please, previous time it was useful, because I do not see something unusual. Thanks ----- Original Message ----- From: "Cong Yue" <Cong_Yue(a)alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> To: "Artyom Lukianov" <alukiano(a)redhat.com<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>> Cc: "Simone Tiraboschi" <stirabos(a)redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>>, users(a)ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org> Sent: Monday, December 29, 2014 7:15:24 AM Subject: Re: [ovirt-users] VM failover with ovirt3.5 Also I change the maintenance mode to local in another host. But also the VM in this host can not be migrated. The logs are as follows. [root@compute2-2 ~]# hosted-engine --set-maintenance --mode=local [root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log MainThread::INFO::2014-12-28 21:09:04,184::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-28 21:09:14,603::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-28 21:09:14,603::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-28 21:09:24,903::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-28 21:09:24,904::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-28 21:09:35,026::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host 10.0.0.94 (id 1) MainThread::INFO::2014-12-28 21:09:35,236::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-28 21:09:35,236::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-28 21:09:45,604::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-28 21:09:45,604::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-28 21:09:55,691::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-28 21:09:55,701::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1419829795.7 type=state_transition detail=EngineDown-LocalMaintenance hostname='compute2-2' MainThread::INFO::2014-12-28 21:09:55,761::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-LocalMaintenance) sent? sent MainThread::INFO::2014-12-28 21:09:55,990::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is 0 due to local maintenance mode MainThread::INFO::2014-12-28 21:09:55,990::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-28 21:09:55,991::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) ^C You have new mail in /var/spool/mail/root [root@compute2-2 ~]# ps -ef | grep qemu root 18420 2777 0 21:10<x-apple-data-detectors://39> pts/0 00:00:00<x-apple-data-detectors://40> grep --color=auto qemu qemu 29809 1 0 Dec19 ? 01:17:20 /usr/libexec/qemu-kvm -name testvm2-2 -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Nehalem -m 500 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid c31e97d0-135e-42da-9954-162b5228dce3 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-0.1406.el7.centos.2.5,serial=4C4C4544-0059-3610-8033-B4C04F395931,uuid=c31e97d0-135e-42da-9954-162b5228dce3 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/testvm2-2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2014-12-19T20:17:17<x-apple-data-detectors://42>,driftfix=slew -no-kvm-pit-reinjection -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/5cbeb8c9-4f04-48d0-a5eb-78c49187c550/a0570e8c-9867-4ec4-818f-11e102fc4f9b,if=none,id=drive-virtio-disk0,format=qcow2,serial=5cbeb8c9-4f04-48d0-a5eb-78c49187c550,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=28,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:db:94:00,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5901,addr=10.0.0.93,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864<tel:67108864> -global qxl-vga.vram_size=33554432<tel:33554432> -incoming tcp:[::]:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 [root@compute2-2 ~]# Thanks, Cong On 2014/12/28, at 20:53, "Yue, Cong" <Cong_Yue(a)alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> wrote: I checked it again and confirmed there is one guest VM is running on the top of this host. The log is as follows: [root@compute2-1 vdsm]# ps -ef | grep qemu qemu 2983 846 0 Dec19 ? 00:00:00<x-apple-data-detectors://0> [supervdsmServer] <defunct> root 5489 3053 0 20:49<x-apple-data-detectors://1> pts/0 00:00:00<x-apple-data-detectors://2> grep --color=auto qemu qemu 26128 1 0 Dec19 ? 01:09:19 /usr/libexec/qemu-kvm -name testvm2 -S -machine rhel6.5.0,accel=kvm,usb=off -cpu Nehalem -m 500 -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -uuid e46bca87-4df5-4287-844b-90a26fccef33 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-0.1406.el7.centos.2.5,serial=4C4C4544-0030-3310-8059-B8C04F585231,uuid=e46bca87-4df5-4287-844b-90a26fccef33 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/testvm2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2014-12-19T20:18:01<x-apple-data-detectors://4>,driftfix=slew -no-kvm-pit-reinjection -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/b4b5426b-95e3-41af-b286-da245891cdaf/0f688d49-97e3-4f1d-84d4-ac1432d903b3,if=none,id=drive-virtio-disk0,format=qcow2,serial=b4b5426b-95e3-41af-b286-da245891cdaf,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:db:94:01,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef33.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef33.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice tls-port=5900,addr=10.0.0.92,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864<tel:67108864> -global qxl-vga.vram_size=33554432<tel:33554432> -incoming tcp:[::]:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 [root@compute2-1 vdsm]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log MainThread::INFO::2014-12-28 20:49:27,315::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-28 20:49:27,646::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-28 20:49:27,646::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-28 20:49:37,732::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-28 20:49:37,961::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-28 20:49:37,961::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-28 20:49:48,048::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-28 20:49:48,319::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Score is 0 due to local maintenance mode MainThread::INFO::2014-12-28 20:49:48,319::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-28 20:49:48,319::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) Thanks, Cong On 2014/12/28, at 3:46, "Artyom Lukianov" <alukiano(a)redhat.com<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>> wrote: I see that you set local maintenance on host3 that do not have engine vm on it, so it nothing to migrate from this host. If you set local maintenance on host1, vm must migrate to another host with positive score. Thanks ----- Original Message ----- From: "Cong Yue" <Cong_Yue(a)alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> To: "Simone Tiraboschi" <stirabos(a)redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>> Cc: users(a)ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org> Sent: Saturday, December 27, 2014 6:58:32 PM Subject: Re: [ovirt-users] VM failover with ovirt3.5 Hi I had a try with "hosted-engine --set-maintence --mode=local" on compute2-1, which is host 3 in my cluster. From the log, it shows maintence mode is dectected, but migration does not happen. The logs are as follows. Is there any other config I need to check? [root@compute2-1 vdsm]# hosted-engine --vm-status --== Host 1 status ==- Status up-to-date : True Hostname : 10.0.0.94 Host ID : 1 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 2400 Local maintenance : False Host timestamp : 836296 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=836296 (Sat Dec 27 11:42:39 2014) host-id=1 score=2400 maintenance=False state=EngineUp --== Host 2 status ==-- Status up-to-date : True Hostname : 10.0.0.93 Host ID : 2 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 2400 Local maintenance : False Host timestamp : 687358 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=687358 (Sat Dec 27 08:42:04 2014) host-id=2 score=2400 maintenance=False state=EngineDown --== Host 3 status ==-- Status up-to-date : True Hostname : 10.0.0.92 Host ID : 3 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 0 Local maintenance : True Host timestamp : 681827 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=681827 (Sat Dec 27 08:42:40 2014) host-id=3 score=0 maintenance=True state=LocalMaintenance [root@compute2-1 vdsm]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log MainThread::INFO::2014-12-27 08:42:41,109::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-27 08:42:51,198::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-27 08:42:51,420::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-27 08:42:51,420::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-27 08:43:01,507::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-27 08:43:01,773::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-27 08:43:01,773::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-27 08:43:11,859::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Local maintenance detected MainThread::INFO::2014-12-27 08:43:12,072::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state LocalMaintenance (score: 0) MainThread::INFO::2014-12-27 08:43:12,072::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) [root@compute2-3 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log MainThread::INFO::2014-12-27 11:36:28,855::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-27 11:36:39,130::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-27 11:36:39,130::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-27 11:36:49,449::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-27 11:36:49,449::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-27 11:36:59,739::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-27 11:36:59,739::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-27 11:37:09,779::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost MainThread::INFO::2014-12-27 11:37:10,026::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-27 11:37:10,026::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-27 11:37:20,331::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-27 11:37:20,331::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) [root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.log MainThread::INFO::2014-12-27 08:36:12,462::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-27 08:36:22,797::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-27 08:36:22,798::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-27 08:36:32,876::states::437::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host 10.0.0.94 (id 1) MainThread::INFO::2014-12-27 08:36:33,169::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-27 08:36:33,169::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-27 08:36:43,567::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-27 08:36:43,567::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-27 08:36:53,858::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-27 08:36:53,858::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-27 08:37:04,028::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2014-12-27 08:37:04,028::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host 10.0.0.94 (id 1): {'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=835987 (Sat Dec 27 11:37:30 2014)\nhost-id=1\nscore=2400\nmaintenance=False\nstate=EngineUp\n', 'hostname': '10.0.0.94', 'alive': True, 'host-id': 1, 'engine-status': {'health': 'good', 'vm': 'up', 'detail': 'up'}, 'score': 2400, 'maintenance': False, 'host-ts': 835987} MainThread::INFO::2014-12-27 08:37:04,028::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host 10.0.0.92 (id 3): {'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=681528 (Sat Dec 27 08:37:41 2014)\nhost-id=3\nscore=0\nmaintenance=True\nstate=LocalMaintenance\n', 'hostname': '10.0.0.92', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 0, 'maintenance': True, 'host-ts': 681528} MainThread::INFO::2014-12-27 08:37:04,028::state_machine::168::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 2): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 15300.0, 'maintenance': False, 'cpu-load': 0.0215, 'gateway': True} MainThread::INFO::2014-12-27 08:37:04,265::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-27 08:37:04,265::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) Thanks, Cong On 2014/12/22, at 5:29, "Simone Tiraboschi" <stirabos(a)redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>> wrote: ----- Original Message ----- From: "Cong Yue" <Cong_Yue(a)alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> To: "Simone Tiraboschi" <stirabos(a)redhat.com<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>> Cc: users(a)ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org> Sent: Friday, December 19, 2014 7:22:10 PM Subject: RE: [ovirt-users] VM failover with ovirt3.5 Thanks for the information. This is the log for my three ovirt nodes. >From the output of hosted-engine --vm-status, it shows the engine state for my 2nd and 3rd ovirt node is DOWN. Is this the reason why VM failover not work in my environment? No, they looks ok: you can run the engine VM on single host at a time. How can I make also engine works for my 2nd and 3rd ovit nodes? If you put the host 1 in local maintenance mode ( hosted-engine --set-maintenance --mode=local ) the VM should migrate to host 2; if you reactivate host 1 ( hosted-engine --set-maintenance --mode=none ) and put host 2 in local maintenance mode the VM should migrate again. Can you please try that and post the logs if something is going bad? -- --== Host 1 status ==-- Status up-to-date : True Hostname : 10.0.0.94 Host ID : 1 Engine status : {"health": "good", "vm": "up", "detail": "up"} Score : 2400 Local maintenance : False Host timestamp : 150475 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=150475 (Fri Dec 19 13:12:18 2014) host-id=1 score=2400 maintenance=False state=EngineUp --== Host 2 status ==-- Status up-to-date : True Hostname : 10.0.0.93 Host ID : 2 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 2400 Local maintenance : False Host timestamp : 1572 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1572 (Fri Dec 19 10:12:18 2014) host-id=2 score=2400 maintenance=False state=EngineDown --== Host 3 status ==-- Status up-to-date : False Hostname : 10.0.0.92 Host ID : 3 Engine status : unknown stale-data Score : 2400 Local maintenance : False Host timestamp : 987 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=987 (Fri Dec 19 10:09:58 2014) host-id=3 score=2400 maintenance=False state=EngineDown -- And the /var/log/ovirt-hosted-engine-ha/agent.log for three ovirt nodes are as follows: -- 10.0.0.94(hosted-engine-1) --- MainThread::INFO::2014-12-19 13:09:33,716::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:09:33,716::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:09:44,017::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:09:44,017::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:09:54,303::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:09:54,303::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:10:04,342::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost MainThread::INFO::2014-12-19 13:10:04,617::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:10:04,617::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:10:14,657::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2014-12-19 13:10:14,657::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host 10.0.0.93 (id 2): {'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1448 (Fri Dec 19 10:10:14 2014)\nhost-id=2\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 'hostname': '10.0.0.93', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 'host-ts': 1448} MainThread::INFO::2014-12-19 13:10:14,657::state_machine::165::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host 10.0.0.92 (id 3): {'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=987 (Fri Dec 19 10:09:58 2014)\nhost-id=3\nscore=2400\nmaintenance=False\nstate=EngineDown\n', 'hostname': '10.0.0.92', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 2400, 'maintenance': False, 'host-ts': 987} MainThread::INFO::2014-12-19 13:10:14,658::state_machine::168::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'health': 'good', 'vm': 'up', 'detail': 'up'}, 'bridge': True, 'mem-free': 1079.0, 'maintenance': False, 'cpu-load': 0.0269, 'gateway': True} MainThread::INFO::2014-12-19 13:10:14,904::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:10:14,904::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:10:25,210::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:10:25,210::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:10:35,499::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:10:35,499::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:10:45,784::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:10:45,785::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:10:56,070::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:10:56,070::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:11:06,109::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost MainThread::INFO::2014-12-19 13:11:06,359::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:11:06,359::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:11:16,658::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:11:16,658::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:11:26,991::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:11:26,991::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) MainThread::INFO::2014-12-19 13:11:37,341::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineUp (score: 2400) MainThread::INFO::2014-12-19 13:11:37,341::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.93 (id: 2, score: 2400) ---- 10.0.0.93 (hosted-engine-2) MainThread::INFO::2014-12-19 10:12:18,339::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-19 10:12:18,339::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-19 10:12:28,651::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-19 10:12:28,652::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-19 10:12:39,010::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-19 10:12:39,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-19 10:12:49,338::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-19 10:12:49,338::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-19 10:12:59,642::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-19 10:12:59,642::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) MainThread::INFO::2014-12-19 10:13:10,010::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-12-19 10:13:10,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host 10.0.0.94 (id: 1, score: 2400) 10.0.0.92(hosted-engine-3) same as 10.0.0.93 -- -----Original Message----- From: Simone Tiraboschi [mailto:stirabos@redhat.com] Sent: Friday, December 19, 2014 12:28 AM To: Yue, Cong Cc: users(a)ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org> Subject: Re: [ovirt-users] VM failover with ovirt3.5 ----- Original Message ----- From: "Cong Yue" <Cong_Yue(a)alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>> To: users(a)ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org> Sent: Friday, December 19, 2014 2:14:33 AM Subject: [ovirt-users] VM failover with ovirt3.5 Hi In my environment, I have 3 ovirt nodes as one cluster. And on top of host-1, there is one vm to host ovirt engine. Also I have one external storage for the cluster to use as data domain of engine and data. I confirmed live migration works well in my environment. But it seems very buggy for VM failover if I try to force to shut down one ovirt node. Sometimes the VM in the node which is shutdown can migrate to other host, but it take more than several minutes. Sometimes, it can not migrate at all. Sometimes, only when the host is back, the VM is beginning to move. Can you please check or share the logs under /var/log/ovirt-hosted-engine-ha/ ? Is there some documentation to explain how VM failover is working? And is there some bugs reported related with this? http://www.ovirt.org/Features/Self_Hosted_Engine#Agent_State_Diagram Thanks in advance, Cong This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. _______________________________________________ Users mailing list Users(a)ovirt.org<mailto:Users@ovirt.org><mailto:Users@ovirt.org><mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. _______________________________________________ Users mailing list Users(a)ovirt.org<mailto:Users@ovirt.org><mailto:Users@ovirt.org><mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users ________________________________ This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. ________________________________ This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. ________________________________ This e-mail message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's e-mail System Administrator. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ovirt.org/pipermail/users/attachments/20141229/4ec6cc13/atta...> ------------------------------ _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users End of Users Digest, Vol 39, Issue 171 ************************************** ------=_Part_1882534_1428653136.1419879250032 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: georgia,serif; font-size: 12pt; colo= r: #000000"><div>Hi,</div><div>Can you please provide engine.log from = /var/log/ovirt-engine/engine.log and to try as follows:</div><div><ol><li>R= evert all tree hosts to maintenance-mode=3Dnone.</li><li>Check that engine = up and running.</li><li>Turn one of the hosts that is not running the engin= e, to maintenance mode local.</li><li>Turn host that is running the engine = to maintenance mode local.</li><li>Check that engine migrated to one and th= e only remaining host, that had not been put in to maintenance mode at all.= </li></ol><div>Can you also provide your engine version, is it 3.4 somethin= g?</div></div><div><br></div><div><span name=3D"x"></span><br>Thanks in adv= ance.<br><div><br></div>Best regards,<br>Nikolai<br>____________________<br= >Nikolai Sednev<br>Senior Quality Engineer at Compute team<br>Red Hat Israe= l<br>34 Jerusalem Road,<br>Ra'anana, Israel 43501<br><div><br></div>Tel: &n= bsp;     +972   9 7692043<br>Mobile: +972 52 7342734<br>Emai= l: nsednev(a)redhat.com<br>IRC: nsednev<span name=3D"x"></span><br></div><div= ><br></div><hr id=3D"zwchr"><div style=3D"color:#000;font-weight:normal;fon= t-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;= font-size:12pt;"><b>From: </b>users-request(a)ovirt.org<br><b>To: </b>users@o= virt.org<br><b>Sent: </b>Monday, December 29, 2014 8:29:36 PM<br><b>Subject= : </b>Users Digest, Vol 39, Issue 171<br><div><br></div>Send Users mailing = list submissions to<br>        user= s(a)ovirt.org<br><div><br></div>To subscribe or unsubscribe via the World Wid= e Web, visit<br>        http://list= s.ovirt.org/mailman/listinfo/users<br>or, via email, send a message with su= bject or body 'help' to<br>        = users-request(a)ovirt.org<br><div><br></div>You can reach the person managing= the list at<br>        users-owner= @ovirt.org<br><div><br></div>When replying, please edit your Subject line s= o it is more specific<br>than "Re: Contents of Users digest..."<br><div><br= ></div><br>Today's Topics:<br><div><br></div>   1. Re:  VM f= ailover with ovirt3.5 (Yue, Cong)<br><div><br></div><br>-------------------= ---------------------------------------------------<br><div><br></div>Messa= ge: 1<br>Date: Mon, 29 Dec 2014 10:29:04 -0800<br>From: "Yue, Cong" <Con= g_Yue(a)alliedtelesis.com><br>To: Artyom Lukianov <alukiano(a)redhat.com&= gt;<br>Cc: "users(a)ovirt.org" <users(a)ovirt.org><br>Subject: Re: [ovirt= -users] VM failover with ovirt3.5<br>Message-ID: <21D302CF-AD6F-4E8C-A37= 3-52ADAC1C129B(a)alliedtelesis.com><br>Content-Type: text/plain; charset= =3D"utf-8"<br><div><br></div>I disabled local maintenance mode for all host= s, and then only set the host where HE VM is there to local maintenance mod= e. The logs are as follows. During the migration of HE VM , it shows some f= atal error happen. By the way, also HE VM can not work with live migration.= Instead, other VMs can do live migration.<br><div><br></div>---<br>[root@c= ompute2-3 ~]# hosted-engine --set-maintenance --mode=3Dlocal<br>You have ne= w mail in /var/spool/mail/root<br>[root@compute2-3 ~]# tail -f /var/log/ovi= rt-hosted-engine-ha/agent.log<br>MainThread::INFO::2014-12-29<br>13:16:12,4= 35::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEn= gine::(start_monitoring)<br>Best remote host 10.0.0.92 (id: 3, score: 2400)= <br>MainThread::INFO::2014-12-29<br>13:16:22,711::hosted_engine::327::ovirt= _hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>C= urrent state EngineUp (score: 2400)<br>MainThread::INFO::2014-12-29<br>13:1= 6:22,711::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Ho= stedEngine::(start_monitoring)<br>Best remote host 10.0.0.92 (id: 3, score:= 2400)<br>MainThread::INFO::2014-12-29<br>13:16:32,978::hosted_engine::327:= :ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring= )<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::2014-12-29<b= r>13:16:32,978::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2, = score: 2400)<br>MainThread::INFO::2014-12-29<br>13:16:43,272::hosted_engine= ::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni= toring)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::2014-1= 2-29<br>13:16:43,272::hosted_engine::332::ovirt_hosted_engine_ha.agent.host= ed_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (i= d: 2, score: 2400)<br>MainThread::INFO::2014-12-29<br>13:16:53,316::states:= :394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)<br= >Engine vm running on localhost<br>MainThread::INFO::2014-12-29<br>13:16:53= ,562::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted= Engine::(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainT= hread::INFO::2014-12-29<br>13:16:53,562::hosted_engine::332::ovirt_hosted_e= ngine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remot= e host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-29<br>13= :17:03,600::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(check)<br>Local maintenance detected<br>MainThread::INFO:= :2014-12-29<br>13:17:03,611::brokerlink::111::ovirt_hosted_engine_ha.lib.br= okerlink.BrokerLink::(notify)<br>Trying: notify time=3D1419877023.61 type= =3Dstate_transition<br>detail=3DEngineUp-LocalMaintenanceMigrateVm hostname= =3D'compute2-3'<br>MainThread::INFO::2014-12-29<br>13:17:03,672::brokerlink= ::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)<br>Succes= s, was notification of state_transition<br>(EngineUp-LocalMaintenanceMigrat= eVm) sent? sent<br>MainThread::INFO::2014-12-29<br>13:17:03,911::states::20= 8::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score)<br>Scor= e is 0 due to local maintenance mode<br>MainThread::INFO::2014-12-29<br>13:= 17:03,912::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.H= ostedEngine::(start_monitoring)<br>Current state LocalMaintenanceMigrateVm = (score: 0)<br>MainThread::INFO::2014-12-29<br>13:17:03,912::hosted_engine::= 332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monito= ring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INF= O::2014-12-29<br>13:17:03,960::brokerlink::111::ovirt_hosted_engine_ha.lib.= brokerlink.BrokerLink::(notify)<br>Trying: notify time=3D1419877023.96 type= =3Dstate_transition<br>detail=3DLocalMaintenanceMigrateVm-EngineMigratingAw= ay<br>hostname=3D'compute2-3'<br>MainThread::INFO::2014-12-29<br>13:17:03,9= 80::brokerlink::120::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(not= ify)<br>Success, was notification of state_transition<br>(LocalMaintenanceM= igrateVm-EngineMigratingAway) sent? sent<br>MainThread::INFO::2014-12-29<br= >13:17:04,218::states::66::ovirt_hosted_engine_ha.agent.hosted_engine.Hoste= dEngine::(_penalize_memory)<br>Penalizing score by 400 due to low free memo= ry<br>MainThread::INFO::2014-12-29<br>13:17:04,218::hosted_engine::327::ovi= rt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br= >Current state EngineMigratingAway (score: 2000)<br>MainThread::INFO::2014-= 12-29<br>13:17:04,219::hosted_engine::332::ovirt_hosted_engine_ha.agent.hos= ted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (= id: 2, score: 2400)<br>MainThread::ERROR::2014-12-29<br>13:17:14,251::hoste= d_engine::867::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_m= onitor_migration)<br>Failed to migrate<br>Traceback (most recent call last)= :<br> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/ag= ent/hosted_engine.py",<br>line 863, in _monitor_migration<br>   v= m_id,<br> File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_h= a/lib/vds_client.py",<br>line 85, in run_vds_client_cmd<br>   res= ponse['status']['message'])<br>DetailedError: Error 12 from migrateStatus: = Fatal error during migration<br>MainThread::INFO::2014-12-29<br>13:17:14,26= 2::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(noti= fy)<br>Trying: notify time=3D1419877034.26 type=3Dstate_transition<br>detai= l=3DEngineMigratingAway-ReinitializeFSM hostname=3D'compute2-3'<br>MainThre= ad::INFO::2014-12-29<br>13:17:14,263::brokerlink::120::ovirt_hosted_engine_= ha.lib.brokerlink.BrokerLink::(notify)<br>Success, was notification of stat= e_transition<br>(EngineMigratingAway-ReinitializeFSM) sent? ignored<br>Main= Thread::INFO::2014-12-29<br>13:17:14,496::hosted_engine::327::ovirt_hosted_= engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current s= tate ReinitializeFSM (score: 0)<br>MainThread::INFO::2014-12-29<br>13:17:14= ,496::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted= Engine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2, score: 240= 0)<br>MainThread::INFO::2014-12-29<br>13:17:24,536::state_decorators::124::= ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>Local m= aintenance detected<br>MainThread::INFO::2014-12-29<br>13:17:24,547::broker= link::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)<br>Tr= ying: notify time=3D1419877044.55 type=3Dstate_transition<br>detail=3DReini= tializeFSM-LocalMaintenance hostname=3D'compute2-3'<br>MainThread::INFO::20= 14-12-29<br>13:17:24,574::brokerlink::120::ovirt_hosted_engine_ha.lib.broke= rlink.BrokerLink::(notify)<br>Success, was notification of state_transition= <br>(ReinitializeFSM-LocalMaintenance) sent? sent<br>MainThread::INFO::2014= -12-29<br>13:17:24,812::hosted_engine::327::ovirt_hosted_engine_ha.agent.ho= sted_engine.HostedEngine::(start_monitoring)<br>Current state LocalMaintena= nce (score: 0)<br>MainThread::INFO::2014-12-29<br>13:17:24,812::hosted_engi= ne::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_mo= nitoring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread:= :INFO::2014-12-29<br>13:17:34,851::state_decorators::124::ovirt_hosted_engi= ne_ha.agent.hosted_engine.HostedEngine::(check)<br>Local maintenance detect= ed<br>MainThread::INFO::2014-12-29<br>13:17:35,095::hosted_engine::327::ovi= rt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br= >Current state LocalMaintenance (score: 0)<br>MainThread::INFO::2014-12-29<= br>13:17:35,095::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_en= gine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2,= score: 2400)<br>MainThread::INFO::2014-12-29<br>13:17:45,130::state_decora= tors::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)= <br>Local maintenance detected<br>MainThread::INFO::2014-12-29<br>13:17:45,= 368::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedE= ngine::(start_monitoring)<br>Current state LocalMaintenance (score: 0)<br>M= ainThread::INFO::2014-12-29<br>13:17:45,368::hosted_engine::332::ovirt_host= ed_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best r= emote host 10.0.0.93 (id: 2, score: 2400)<br>^C<br>[root@compute2-3 ~]#<br>= <div><br></div><br>[root@compute2-3 ~]# hosted-engine --vm-status<br><div><= br></div><br>--=3D=3D Host 1 status =3D=3D--<br><div><br></div>Status up-to= -date                  : True<= br>Hostname                   =         : 10.0.0.94<br>Host ID       &nb= sp;                    : = 1<br>Engine status                 =      : {"health": "good", "vm": "up",<br>"detail": "up"}<br>= Score                    =          : 0<br>Local maintenance     &= nbsp;            : True<br>Host timestamp &nb= sp;                   : 101495= 6<tel:1014956><br>Extra metadata (valid at timestamp):<br>metadata_pa= rse_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D1014956<t= el:1014956> (Mon Dec 29 13:20:19 2014)<br>host-id=3D1<br>score=3D0<br>ma= intenance=3DTrue<br>state=3DLocalMaintenance<br><div><br></div><br>--=3D=3D= Host 2 status =3D=3D--<br><div><br></div>Status up-to-date     &= nbsp;            : True<br>Hostname   &n= bsp;                     =   : 10.0.0.93<br>Host ID             &nb= sp;              : 2<br>Engine status &n= bsp;                    := {"reason": "vm not running on<br>this host", "health": "bad", "vm": "down"= , "detail": "unknown"}<br>Score             &= nbsp;                : 2400<br>Loca= l maintenance                 &nbsp= ;: False<br>Host timestamp              =       : 866019<br>Extra metadata (valid at timestamp):<br>m= etadata_parse_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D86= 6019 (Mon Dec 29 10:19:45 2014)<br>host-id=3D2<br>score=3D2400<br>maintenan= ce=3DFalse<br>state=3DEngineDown<br><div><br></div><br>--=3D=3D Host 3 stat= us =3D=3D--<br><div><br></div>Status up-to-date        =          : True<br>Hostname       =                     : 10.= 0.0.92<br>Host ID                 &= nbsp;          : 3<br>Engine status     =                  : {"reason": = "vm not running on<br>this host", "health": "bad", "vm": "down", "detail": = "unknown"}<br>Score                =              : 2400<br>Local maintenanc= e                  : False<br>= Host timestamp                 &nbs= p;   : 860493<br>Extra metadata (valid at timestamp):<br>metadata_pars= e_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D860493 (Mon De= c 29 10:20:35 2014)<br>host-id=3D3<br>score=3D2400<br>maintenance=3DFalse<b= r>state=3DEngineDown<br>[root@compute2-3 ~]#<br>---<br>Thanks,<br>Cong<br><= div><br></div><br><div><br></div>On 2014/12/29, at 8:43, "Artyom Lukianov" = <alukiano@redhat.com<mailto:alukiano@redhat.com>> wrote:<br><di= v><br></div>I see that HE vm run on host with ip 10.0.0.94, and two another= hosts in "Local Maintenance" state, so vm will not migrate to any of them,= can you try disable local maintenance on all hosts in HE environment and a= fter enable "local maintenance" on host where HE vm run, and provide also o= utput of hosted-engine --vm-status.<br>Failover works in next way:<br>1) if= host where run HE vm have score less by 800 that some other host in HE env= ironment, HE vm will migrate on host with best score<br>2) if something hap= pen to vm(kernel panic, crash of service...), agent will restart HE vm on a= nother host in HE environment with positive score<br>3) if put to local mai= ntenance host with HE vm, vm will migrate to another host with positive sco= re<br>Thanks.<br><div><br></div>----- Original Message -----<br>From: "Cong= Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com&g= t;><br>To: "Artyom Lukianov" <alukiano@redhat.com<mailto:alukiano@= redhat.com>><br>Cc: "Simone Tiraboschi" <stirabos(a)redhat.com<ma= ilto:stirabos@redhat.com>>, users@ovirt.org<mailto:users@ovirt.org= ><br>Sent: Monday, December 29, 2014 6:30:42 PM<br>Subject: Re: [ovirt-u= sers] VM failover with ovirt3.5<br><div><br></div>Thanks and the --vm-statu= s log is as follows:<br>[root@compute2-2 ~]# hosted-engine --vm-status<br><= div><br></div><br>--=3D=3D Host 1 status =3D=3D--<br><div><br></div>Status = up-to-date                  : = True<br>Hostname                 &n= bsp;         : 10.0.0.94<br>Host ID     &nbsp= ;                     &nb= sp;: 1<br>Engine status               &n= bsp;      : {"health": "good", "vm": "up",<br>"detail": "up"= }<br>Score                   &= nbsp;          : 2400<br>Local maintenance   =                : False<br>Host time= stamp                    = : 1008087<br>Extra metadata (valid at timestamp):<br>metadata_parse_versio= n=3D1<br>metadata_feature_version=3D1<br>timestamp=3D1008087<tel:1008087= > (Mon Dec 29 11:25:51 2014)<br>host-id=3D1<br>score=3D2400<br>maintenan= ce=3DFalse<br>state=3DEngineUp<br><div><br></div><br>--=3D=3D Host 2 status= =3D=3D--<br><div><br></div>Status up-to-date         &= nbsp;        : True<br>Hostname       &n= bsp;                   : 10.0.= 0.93<br>Host ID                 &nb= sp;          : 2<br>Engine status     &n= bsp;                : {"reason": "v= m not running on<br>this host", "health": "bad", "vm": "down", "detail": "u= nknown"}<br>Score                 &= nbsp;            : 0<br>Local maintenance &nb= sp;                : True<br>Host t= imestamp                   &nb= sp; : 859142<br>Extra metadata (valid at timestamp):<br>metadata_parse_vers= ion=3D1<br>metadata_feature_version=3D1<br>timestamp=3D859142 (Mon Dec 29 0= 8:25:08 2014)<br>host-id=3D2<br>score=3D0<br>maintenance=3DTrue<br>state=3D= LocalMaintenance<br><div><br></div><br>--=3D=3D Host 3 status =3D=3D--<br><= div><br></div>Status up-to-date             &= nbsp;    : True<br>Hostname           &n= bsp;               : 10.0.0.92<br>Host I= D                     &nb= sp;      : 3<br>Engine status         &n= bsp;            : {"reason": "vm not running = on<br>this host", "health": "bad", "vm": "down", "detail": "unknown"}<br>Sc= ore                     &= nbsp;        : 0<br>Local maintenance     &nb= sp;            : True<br>Host timestamp &nbsp= ;                   : 853615<b= r>Extra metadata (valid at timestamp):<br>metadata_parse_version=3D1<br>met= adata_feature_version=3D1<br>timestamp=3D853615 (Mon Dec 29 08:25:57 2014)<= br>host-id=3D3<br>score=3D0<br>maintenance=3DTrue<br>state=3DLocalMaintenan= ce<br>You have new mail in /var/spool/mail/root<br>[root@compute2-2 ~]#<br>= <div><br></div>Could you please explain how VM failover works inside ovirt?= Is there any other debug option I can enable to check the problem?<br><div= ><br></div>Thanks,<br>Cong<br><div><br></div><br>On 2014/12/29, at 1:39, "A= rtyom Lukianov" <alukiano@redhat.com<mailto:alukiano@redhat.com>&l= t;mailto:alukiano@redhat.com>> wrote:<br><div><br></div>Can you also = provide output of hosted-engine --vm-status please, previous time it was us= eful, because I do not see something unusual.<br>Thanks<br><div><br></div>-= ---- Original Message -----<br>From: "Cong Yue" <Cong_Yue(a)alliedtelesis.= com<mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedteles= is.com>><br>To: "Artyom Lukianov" <alukiano@redhat.com<mailto:a= lukiano@redhat.com><mailto:alukiano@redhat.com>><br>Cc: "Simone= Tiraboschi" <stirabos@redhat.com<mailto:stirabos@redhat.com><m= ailto:stirabos@redhat.com>>, users@ovirt.org<mailto:users@ovirt.or= g><mailto:users@ovirt.org><br>Sent: Monday, December 29, 2014 7:15= :24 AM<br>Subject: Re: [ovirt-users] VM failover with ovirt3.5<br><div><br>= </div>Also I change the maintenance mode to local in another host. But also= the VM in this host can not be migrated. The logs are as follows.<br><div>= <br></div>[root@compute2-2 ~]# hosted-engine --set-maintenance --mode=3Dloc= al<br>[root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.lo= g<br>MainThread::INFO::2014-12-28<br>21:09:04,184::hosted_engine::332::ovir= t_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>= Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-1= 2-28<br>21:09:14,603::hosted_engine::327::ovirt_hosted_engine_ha.agent.host= ed_engine.HostedEngine::(start_monitoring)<br>Current state EngineDown (sco= re: 2400)<br>MainThread::INFO::2014-12-28<br>21:09:14,603::hosted_engine::3= 32::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitor= ing)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO= ::2014-12-28<br>21:09:24,903::hosted_engine::327::ovirt_hosted_engine_ha.ag= ent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state EngineD= own (score: 2400)<br>MainThread::INFO::2014-12-28<br>21:09:24,904::hosted_e= ngine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start= _monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThre= ad::INFO::2014-12-28<br>21:09:35,026::states::437::ovirt_hosted_engine_ha.a= gent.hosted_engine.HostedEngine::(consume)<br>Engine vm is running on host = 10.0.0.94 (id 1)<br>MainThread::INFO::2014-12-28<br>21:09:35,236::hosted_en= gine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_= monitoring)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::= 2014-12-28<br>21:09:35,236::hosted_engine::332::ovirt_hosted_engine_ha.agen= t.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0= .94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-28<br>21:09:45,604::h= osted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:= :(start_monitoring)<br>Current state EngineDown (score: 2400)<br>MainThread= ::INFO::2014-12-28<br>21:09:45,604::hosted_engine::332::ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote hos= t 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-28<br>21:09:5= 5,691::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.Ho= stedEngine::(check)<br>Local maintenance detected<br>MainThread::INFO::2014= -12-28<br>21:09:55,701::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerl= ink.BrokerLink::(notify)<br>Trying: notify time=3D1419829795.7 type=3Dstate= _transition<br>detail=3DEngineDown-LocalMaintenance hostname=3D'compute2-2'= <br>MainThread::INFO::2014-12-28<br>21:09:55,761::brokerlink::120::ovirt_ho= sted_engine_ha.lib.brokerlink.BrokerLink::(notify)<br>Success, was notifica= tion of state_transition<br>(EngineDown-LocalMaintenance) sent? sent<br>Mai= nThread::INFO::2014-12-28<br>21:09:55,990::states::208::ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine::(score)<br>Score is 0 due to local ma= intenance mode<br>MainThread::INFO::2014-12-28<br>21:09:55,990::hosted_engi= ne::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_mo= nitoring)<br>Current state LocalMaintenance (score: 0)<br>MainThread::INFO:= :2014-12-28<br>21:09:55,991::hosted_engine::332::ovirt_hosted_engine_ha.age= nt.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.= 0.94 (id: 1, score: 2400)<br>^C<br>You have new mail in /var/spool/mail/roo= t<br>[root@compute2-2 ~]# ps -ef | grep qemu<br>root     18420 &n= bsp;2777  0 21:10<x-apple-data-detectors://39> pts/0   &nbs= p;00:00:00<x-apple-data-detectors://40> grep --color=3Dauto qemu<br>q= emu     29809     1  0 Dec19 ?     &nbsp= ;  01:17:20 /usr/libexec/qemu-kvm<br>-name testvm2-2 -S -machine rhel6= .5.0,accel=3Dkvm,usb=3Doff -cpu Nehalem<br>-m 500 -realtime mlock=3Doff -sm= p<br>1,maxcpus=3D16,sockets=3D16,cores=3D1,threads=3D1 -uuid<br>c31e97d0-13= 5e-42da-9954-162b5228dce3 -smbios<br>type=3D1,manufacturer=3DoVirt,product= =3DoVirt<br>Node,version=3D7-0.1406.el7.centos.2.5,serial=3D4C4C4544-0059-3= 610-8033-B4C04F395931,uuid=3Dc31e97d0-135e-42da-9954-162b5228dce3<br>-no-us= er-config -nodefaults -chardev<br>socket,id=3Dcharmonitor,path=3D/var/lib/l= ibvirt/qemu/testvm2-2.monitor,server,nowait<br>-mon chardev=3Dcharmonitor,i= d=3Dmonitor,mode=3Dcontrol -rtc<br>base=3D2014-12-19T20:17:17<x-apple-da= ta-detectors://42>,driftfix=3Dslew -no-kvm-pit-reinjection<br>-no-hpet -= no-shutdown -boot strict=3Don -device<br>piix3-usb-uhci,id=3Dusb,bus=3Dpci.= 0,addr=3D0x1.0x2 -device<br>virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0= x4 -device<br>virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpc= i.0,addr=3D0x5<br>-drive if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don,format= =3Draw,serial=3D<br>-device ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-= 1-0,id=3Dide0-1-0<br>-drive file=3D/rhev/data-center/00000002-0002-0002-000= 2-0000000001e4/1dc71096-27c4-4256-b2ac-bd7265525c69/images/5cbeb8c9-4f04-48= d0-a5eb-78c49187c550/a0570e8c-9867-4ec4-818f-11e102fc4f9b,if=3Dnone,id=3Ddr= ive-virtio-disk0,format=3Dqcow2,serial=3D5cbeb8c9-4f04-48d0-a5eb-78c49187c5= 50,cache=3Dnone,werror=3Dstop,rerror=3Dstop,aio=3Dthreads<br>-device virtio= -blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk0,id=3D= virtio-disk0,bootindex=3D1<br>-netdev tap,fd=3D28,id=3Dhostnet0,vhost=3Don,= vhostfd=3D29 -device<br>virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00= :1a:4a:db:94:00,bus=3Dpci.0,addr=3D0x3<br>-chardev socket,id=3Dcharchannel0= ,path=3D/var/lib/libvirt/qemu/channels/c31e97d0-135e-42da-9954-162b5228dce3= .com.redhat.rhevm.vdsm,server,nowait<br>-device virtserialport,bus=3Dvirtio= -serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dchannel0,name=3Dcom.redhat.rh= evm.vdsm<br>-chardev socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/= channels/c31e97d0-135e-42da-9954-162b5228dce3.org.qemu.guest_agent.0,server= ,nowait<br>-device virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dc= harchannel1,id=3Dchannel1,name=3Dorg.qemu.guest_agent.0<br>-chardev spicevm= c,id=3Dcharchannel2,name=3Dvdagent -device<br>virtserialport,bus=3Dvirtio-s= erial0.0,nr=3D3,chardev=3Dcharchannel2,id=3Dchannel2,name=3Dcom.redhat.spic= e.0<br>-spice tls-port=3D5901,addr=3D10.0.0.93,x509-dir=3D/etc/pki/vdsm/lib= virt-spice,tls-channel=3Dmain,tls-channel=3Ddisplay,tls-channel=3Dinputs,tl= s-channel=3Dcursor,tls-channel=3Dplayback,tls-channel=3Drecord,tls-channel= =3Dsmartcard,tls-channel=3Dusbredir,seamless-migration=3Don<br>-k en-us -vg= a qxl -global qxl-vga.ram_size=3D67108864<tel:67108864> -global<br>qx= l-vga.vram_size=3D33554432<tel:33554432> -incoming tcp:[::]:49152 -de= vice<br>virtio-balloon-pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7<br>[root@co= mpute2-2 ~]#<br><div><br></div>Thanks,<br>Cong<br><div><br></div><br>On 201= 4/12/28, at 20:53, "Yue, Cong" <Cong_Yue@alliedtelesis.com<mailto:Con= g_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com><mai= lto:Cong_Yue@alliedtelesis.com>> wrote:<br><div><br></div>I checked i= t again and confirmed there is one guest VM is running on the top of this h= ost. The log is as follows:<br><div><br></div>[root@compute2-1 vdsm]# ps -e= f | grep qemu<br>qemu      2983   846  0 Dec19 ? &= nbsp;      00:00:00<x-apple-data-detectors://0> [super= vdsmServer] <defunct><br>root      5489  3053 &nb= sp;0 20:49<x-apple-data-detectors://1> pts/0    00:00:00&lt= ;x-apple-data-detectors://2> grep --color=3Dauto qemu<br>qemu   &nb= sp; 26128     1  0 Dec19 ?        01:09:= 19 /usr/libexec/qemu-kvm<br>-name testvm2 -S -machine rhel6.5.0,accel=3Dkvm= ,usb=3Doff -cpu Nehalem -m<br>500 -realtime mlock=3Doff -smp 1,maxcpus=3D16= ,sockets=3D16,cores=3D1,threads=3D1<br>-uuid e46bca87-4df5-4287-844b-90a26f= ccef33 -smbios<br>type=3D1,manufacturer=3DoVirt,product=3DoVirt<br>Node,ver= sion=3D7-0.1406.el7.centos.2.5,serial=3D4C4C4544-0030-3310-8059-B8C04F58523= 1,uuid=3De46bca87-4df5-4287-844b-90a26fccef33<br>-no-user-config -nodefault= s -chardev<br>socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/testvm2.= monitor,server,nowait<br>-mon chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcon= trol -rtc<br>base=3D2014-12-19T20:18:01<x-apple-data-detectors://4>,d= riftfix=3Dslew -no-kvm-pit-reinjection<br>-no-hpet -no-shutdown -boot stric= t=3Don -device<br>piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -devic= e<br>virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4 -device<br>virtio-se= rial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x5<br>-driv= e if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don,format=3Draw,serial=3D<br>-de= vice ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0<br>-d= rive file=3D/rhev/data-center/00000002-0002-0002-0002-0000000001e4/1dc71096= -27c4-4256-b2ac-bd7265525c69/images/b4b5426b-95e3-41af-b286-da245891cdaf/0f= 688d49-97e3-4f1d-84d4-ac1432d903b3,if=3Dnone,id=3Ddrive-virtio-disk0,format= =3Dqcow2,serial=3Db4b5426b-95e3-41af-b286-da245891cdaf,cache=3Dnone,werror= =3Dstop,rerror=3Dstop,aio=3Dthreads<br>-device virtio-blk-pci,scsi=3Doff,bu= s=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk0,id=3Dvirtio-disk0,bootindex= =3D1<br>-netdev tap,fd=3D26,id=3Dhostnet0,vhost=3Don,vhostfd=3D27 -device<b= r>virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:db:94:01,bus=3D= pci.0,addr=3D0x3<br>-chardev socket,id=3Dcharchannel0,path=3D/var/lib/libvi= rt/qemu/channels/e46bca87-4df5-4287-844b-90a26fccef33.com.redhat.rhevm.vdsm= ,server,nowait<br>-device virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,char= dev=3Dcharchannel0,id=3Dchannel0,name=3Dcom.redhat.rhevm.vdsm<br>-chardev s= ocket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/e46bca87-4df5= -4287-844b-90a26fccef33.org.qemu.guest_agent.0,server,nowait<br>-device vir= tserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dchann= el1,name=3Dorg.qemu.guest_agent.0<br>-chardev spicevmc,id=3Dcharchannel2,na= me=3Dvdagent -device<br>virtserialport,bus=3Dvirtio-serial0.0,nr=3D3,charde= v=3Dcharchannel2,id=3Dchannel2,name=3Dcom.redhat.spice.0<br>-spice tls-port= =3D5900,addr=3D10.0.0.92,x509-dir=3D/etc/pki/vdsm/libvirt-spice,tls-channel= =3Dmain,tls-channel=3Ddisplay,tls-channel=3Dinputs,tls-channel=3Dcursor,tls= -channel=3Dplayback,tls-channel=3Drecord,tls-channel=3Dsmartcard,tls-channe= l=3Dusbredir,seamless-migration=3Don<br>-k en-us -vga qxl -global qxl-vga.r= am_size=3D67108864<tel:67108864> -global<br>qxl-vga.vram_size=3D33554= 432<tel:33554432> -incoming tcp:[::]:49152 -device<br>virtio-balloon-= pci,id=3Dballoon0,bus=3Dpci.0,addr=3D0x7<br>[root@compute2-1 vdsm]# tail -f= /var/log/ovirt-hosted-engine-ha/agent.log<br>MainThread::INFO::2014-12-28<= br>20:49:27,315::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted= _engine.HostedEngine::(check)<br>Local maintenance detected<br>MainThread::= INFO::2014-12-28<br>20:49:27,646::hosted_engine::327::ovirt_hosted_engine_h= a.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state Loc= alMaintenance (score: 0)<br>MainThread::INFO::2014-12-28<br>20:49:27,646::h= osted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:= :(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>M= ainThread::INFO::2014-12-28<br>20:49:37,732::state_decorators::124::ovirt_h= osted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>Local maintena= nce detected<br>MainThread::INFO::2014-12-28<br>20:49:37,961::hosted_engine= ::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni= toring)<br>Current state LocalMaintenance (score: 0)<br>MainThread::INFO::2= 014-12-28<br>20:49:37,961::hosted_engine::332::ovirt_hosted_engine_ha.agent= .hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.= 94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-28<br>20:49:48,048::st= ate_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin= e::(check)<br>Local maintenance detected<br>MainThread::INFO::2014-12-28<br= >20:49:48,319::states::208::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(score)<br>Score is 0 due to local maintenance mode<br>MainThread= ::INFO::2014-12-28<br>20:49:48,319::hosted_engine::327::ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state L= ocalMaintenance (score: 0)<br>MainThread::INFO::2014-12-28<br>20:49:48,319:= :hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin= e::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br= ><div><br></div>Thanks,<br>Cong<br><div><br></div><br>On 2014/12/28, at 3:4= 6, "Artyom Lukianov" <alukiano@redhat.com<mailto:alukiano@redhat.com&= gt;<mailto:alukiano@redhat.com><mailto:alukiano@redhat.com>>= wrote:<br><div><br></div>I see that you set local maintenance on host3 tha= t do not have engine vm on it, so it nothing to migrate from this host.<br>= If you set local maintenance on host1, vm must migrate to another host with= positive score.<br>Thanks<br><div><br></div>----- Original Message -----<b= r>From: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@allied= telesis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue= @alliedtelesis.com>><br>To: "Simone Tiraboschi" <stirabos(a)redhat.c= om<mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><m= ailto:stirabos@redhat.com>><br>Cc: users@ovirt.org<mailto:users@ov= irt.org><mailto:users@ovirt.org><mailto:users@ovirt.org><br>= Sent: Saturday, December 27, 2014 6:58:32 PM<br>Subject: Re: [ovirt-users] = VM failover with ovirt3.5<br><div><br></div>Hi<br><div><br></div>I had a tr= y with "hosted-engine --set-maintence --mode=3Dlocal" on<br>compute2-1, whi= ch is host 3 in my cluster. From the log, it shows<br>maintence mode is dec= tected, but migration does not happen.<br><div><br></div>The logs are as fo= llows. Is there any other config I need to check?<br><div><br></div>[root@c= ompute2-1 vdsm]# hosted-engine --vm-status<br><div><br></div><br>--=3D=3D H= ost 1 status =3D=3D-<br><div><br></div>Status up-to-date     &nbs= p;            : True<br>Hostname   &nbsp= ;                     &nb= sp; : 10.0.0.94<br>Host ID              =              : 1<br>Engine status &nbsp= ;                    : {"= health": "good", "vm": "up",<br>"detail": "up"}<br>Score     &nbs= p;                     &n= bsp;  : 2400<br>Local maintenance           &= nbsp;      : False<br>Host timestamp       &n= bsp;             : 836296<br>Extra metadata (= valid at timestamp):<br>metadata_parse_version=3D1<br>metadata_feature_vers= ion=3D1<br>timestamp=3D836296 (Sat Dec 27 11:42:39 2014)<br>host-id=3D1<br>= score=3D2400<br>maintenance=3DFalse<br>state=3DEngineUp<br><div><br></div><= br>--=3D=3D Host 2 status =3D=3D--<br><div><br></div>Status up-to-date &nbs= p;                : True<br>Hostnam= e                     &nb= sp;     : 10.0.0.93<br>Host ID          =                  : 2<br>Engin= e status                   &nb= sp;  : {"reason": "vm not running on<br>this host", "health": "bad", "= vm": "down", "detail": "unknown"}<br>Score         &nbs= p;                    : 2= 400<br>Local maintenance               &= nbsp;  : False<br>Host timestamp           &n= bsp;         : 687358<br>Extra metadata (valid at times= tamp):<br>metadata_parse_version=3D1<br>metadata_feature_version=3D1<br>tim= estamp=3D687358 (Sat Dec 27 08:42:04 2014)<br>host-id=3D2<br>score=3D2400<b= r>maintenance=3DFalse<br>state=3DEngineDown<br><div><br></div><br>--=3D=3D = Host 3 status =3D=3D--<br><div><br></div>Status up-to-date     &n= bsp;            : True<br>Hostname   &nb= sp;                     &= nbsp; : 10.0.0.92<br>Host ID             &nbs= p;              : 3<br>Engine status &nb= sp;                    : = {"reason": "vm not running on<br>this host", "health": "bad", "vm": "down",= "detail": "unknown"}<br>Score             &n= bsp;                : 0<br>Local ma= intenance                  : T= rue<br>Host timestamp               &nbs= p;     : 681827<br>Extra metadata (valid at timestamp):<br>metada= ta_parse_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D681827 = (Sat Dec 27 08:42:40 2014)<br>host-id=3D3<br>score=3D0<br>maintenance=3DTru= e<br>state=3DLocalMaintenance<br>[root@compute2-1 vdsm]# tail -f /var/log/o= virt-hosted-engine-ha/agent.log<br>MainThread::INFO::2014-12-27<br>08:42:41= ,109::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Hosted= Engine::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 240= 0)<br>MainThread::INFO::2014-12-27<br>08:42:51,198::state_decorators::124::= ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)<br>Local m= aintenance detected<br>MainThread::INFO::2014-12-27<br>08:42:51,420::hosted= _engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(sta= rt_monitoring)<br>Current state LocalMaintenance (score: 0)<br>MainThread::= INFO::2014-12-27<br>08:42:51,420::hosted_engine::332::ovirt_hosted_engine_h= a.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host = 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-27<br>08:43:01,= 507::state_decorators::124::ovirt_hosted_engine_ha.agent.hosted_engine.Host= edEngine::(check)<br>Local maintenance detected<br>MainThread::INFO::2014-1= 2-27<br>08:43:01,773::hosted_engine::327::ovirt_hosted_engine_ha.agent.host= ed_engine.HostedEngine::(start_monitoring)<br>Current state LocalMaintenanc= e (score: 0)<br>MainThread::INFO::2014-12-27<br>08:43:01,773::hosted_engine= ::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni= toring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::I= NFO::2014-12-27<br>08:43:11,859::state_decorators::124::ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine::(check)<br>Local maintenance detected= <br>MainThread::INFO::2014-12-27<br>08:43:12,072::hosted_engine::327::ovirt= _hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>C= urrent state LocalMaintenance (score: 0)<br>MainThread::INFO::2014-12-27<br= >08:43:12,072::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engi= ne.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, s= core: 2400)<br><div><br></div><br><div><br></div>[root@compute2-3 ~]# tail = -f /var/log/ovirt-hosted-engine-ha/agent.log<br>MainThread::INFO::2014-12-2= 7<br>11:36:28,855::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_= engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: = 2, score: 2400)<br>MainThread::INFO::2014-12-27<br>11:36:39,130::hosted_eng= ine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_m= onitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::201= 4-12-27<br>11:36:39,130::hosted_engine::332::ovirt_hosted_engine_ha.agent.h= osted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93= (id: 2, score: 2400)<br>MainThread::INFO::2014-12-27<br>11:36:49,449::host= ed_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(s= tart_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INF= O::2014-12-27<br>11:36:49,449::hosted_engine::332::ovirt_hosted_engine_ha.a= gent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.= 0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-27<br>11:36:59,739= ::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngi= ne::(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThrea= d::INFO::2014-12-27<br>11:36:59,739::hosted_engine::332::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote ho= st 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-27<br>11:37:= 09,779::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin= e::(consume)<br>Engine vm running on localhost<br>MainThread::INFO::2014-12= -27<br>11:37:10,026::hosted_engine::327::ovirt_hosted_engine_ha.agent.hoste= d_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (score:= 2400)<br>MainThread::INFO::2014-12-27<br>11:37:10,026::hosted_engine::332:= :ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring= )<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2= 014-12-27<br>11:37:20,331::hosted_engine::327::ovirt_hosted_engine_ha.agent= .hosted_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (= score: 2400)<br>MainThread::INFO::2014-12-27<br>11:37:20,331::hosted_engine= ::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni= toring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br><div><br></di= v><br>[root@compute2-2 ~]# tail -f /var/log/ovirt-hosted-engine-ha/agent.lo= g<br>MainThread::INFO::2014-12-27<br>08:36:12,462::hosted_engine::332::ovir= t_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>= Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-1= 2-27<br>08:36:22,797::hosted_engine::327::ovirt_hosted_engine_ha.agent.host= ed_engine.HostedEngine::(start_monitoring)<br>Current state EngineDown (sco= re: 2400)<br>MainThread::INFO::2014-12-27<br>08:36:22,798::hosted_engine::3= 32::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitor= ing)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO= ::2014-12-27<br>08:36:32,876::states::437::ovirt_hosted_engine_ha.agent.hos= ted_engine.HostedEngine::(consume)<br>Engine vm is running on host 10.0.0.9= 4 (id 1)<br>MainThread::INFO::2014-12-27<br>08:36:33,169::hosted_engine::32= 7::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitori= ng)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::2014-12-= 27<br>08:36:33,169::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted= _engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.94 (id:= 1, score: 2400)<br>MainThread::INFO::2014-12-27<br>08:36:43,567::hosted_en= gine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_= monitoring)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::= 2014-12-27<br>08:36:43,567::hosted_engine::332::ovirt_hosted_engine_ha.agen= t.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0= .94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-27<br>08:36:53,858::h= osted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:= :(start_monitoring)<br>Current state EngineDown (score: 2400)<br>MainThread= ::INFO::2014-12-27<br>08:36:53,858::hosted_engine::332::ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote hos= t 10.0.0.94 (id: 1, score: 2400)<br>MainThread::INFO::2014-12-27<br>08:37:0= 4,028::state_machine::160::ovirt_hosted_engine_ha.agent.hosted_engine.Hoste= dEngine::(refresh)<br>Global metadata: {'maintenance': False}<br>MainThread= ::INFO::2014-12-27<br>08:37:04,028::state_machine::165::ovirt_hosted_engine= _ha.agent.hosted_engine.HostedEngine::(refresh)<br>Host 10.0.0.94 (id 1): {= 'extra':<br>'metadata_parse_version=3D1\nmetadata_feature_version=3D1\ntime= stamp=3D835987<br>(Sat Dec 27 11:37:30<br>2014)\nhost-id=3D1\nscore=3D2400\= nmaintenance=3DFalse\nstate=3DEngineUp\n',<br>'hostname': '10.0.0.94', 'ali= ve': True, 'host-id': 1, 'engine-status':<br>{'health': 'good', 'vm': 'up',= 'detail': 'up'}, 'score': 2400,<br>'maintenance': False, 'host-ts': 835987= }<br>MainThread::INFO::2014-12-27<br>08:37:04,028::state_machine::165::ovir= t_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Host 10.0= .0.92 (id 3): {'extra':<br>'metadata_parse_version=3D1\nmetadata_feature_ve= rsion=3D1\ntimestamp=3D681528<br>(Sat Dec 27 08:37:41<br>2014)\nhost-id=3D3= \nscore=3D0\nmaintenance=3DTrue\nstate=3DLocalMaintenance\n',<br>'hostname'= : '10.0.0.92', 'alive': True, 'host-id': 3, 'engine-status':<br>{'reason': = 'vm not running on this host', 'health': 'bad', 'vm':<br>'down', 'detail': = 'unknown'}, 'score': 0, 'maintenance': True,<br>'host-ts': 681528}<br>MainT= hread::INFO::2014-12-27<br>08:37:04,028::state_machine::168::ovirt_hosted_e= ngine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Local (id 2): {'eng= ine-health': {'reason': 'vm not running on this<br>host', 'health': 'bad', = 'vm': 'down', 'detail': 'unknown'}, 'bridge':<br>True, 'mem-free': 15300.0,= 'maintenance': False, 'cpu-load': 0.0215,<br>'gateway': True}<br>MainThrea= d::INFO::2014-12-27<br>08:37:04,265::hosted_engine::327::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state = EngineDown (score: 2400)<br>MainThread::INFO::2014-12-27<br>08:37:04,265::h= osted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:= :(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br><= div><br></div>Thanks,<br>Cong<br><div><br></div>On 2014/12/22, at 5:29, "Si= mone Tiraboschi" <stirabos@redhat.com<mailto:stirabos@redhat.com>&= lt;mailto:stirabos@redhat.com><mailto:stirabos@redhat.com>> wro= te:<br><div><br></div><br><div><br></div>----- Original Message -----<br>Fr= om: "Cong Yue" <Cong_Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtele= sis.com><mailto:Cong_Yue@alliedtelesis.com><mailto:Cong_Yue@all= iedtelesis.com>><br>To: "Simone Tiraboschi" <stirabos(a)redhat.com&l= t;mailto:stirabos@redhat.com><mailto:stirabos@redhat.com><mailt= o:stirabos@redhat.com>><br>Cc: users@ovirt.org<mailto:users@ovirt.= org><mailto:users@ovirt.org><mailto:users@ovirt.org><br>Sent= : Friday, December 19, 2014 7:22:10 PM<br>Subject: RE: [ovirt-users] VM fai= lover with ovirt3.5<br><div><br></div>Thanks for the information. This is t= he log for my three ovirt nodes.<br>From the output of hosted-engine --vm-s= tatus, it shows the engine state for<br>my 2nd and 3rd ovirt node is DOWN.<= br>Is this the reason why VM failover not work in my environment?<br><div><= br></div>No, they looks ok: you can run the engine VM on single host at a t= ime.<br><div><br></div>How can I make<br>also engine works for my 2nd and 3= rd ovit nodes?<br><div><br></div>If you put the host 1 in local maintenance= mode ( hosted-engine --set-maintenance --mode=3Dlocal ) the VM should migr= ate to host 2; if you reactivate host 1 ( hosted-engine --set-maintenance -= -mode=3Dnone ) and put host 2 in local maintenance mode the VM should migra= te again.<br><div><br></div>Can you please try that and post the logs if so= mething is going bad?<br><div><br></div><br>--<br>--=3D=3D Host 1 status = =3D=3D--<br><div><br></div>Status up-to-date         &n= bsp;        : True<br>Hostname       &nb= sp;                   : 10.0.0= .94<br>Host ID                 &nbs= p;          : 1<br>Engine status     &nb= sp;                : {"health": "go= od", "vm": "up",<br>"detail": "up"}<br>Score         &n= bsp;                    := 2400<br>Local maintenance              =    : False<br>Host timestamp           =           : 150475<br>Extra metadata (valid at tim= estamp):<br>metadata_parse_version=3D1<br>metadata_feature_version=3D1<br>t= imestamp=3D150475 (Fri Dec 19 13:12:18 2014)<br>host-id=3D1<br>score=3D2400= <br>maintenance=3DFalse<br>state=3DEngineUp<br><div><br></div><br>--=3D=3D = Host 2 status =3D=3D--<br><div><br></div>Status up-to-date     &n= bsp;            : True<br>Hostname   &nb= sp;                     &= nbsp; : 10.0.0.93<br>Host ID             &nbs= p;              : 2<br>Engine status &nb= sp;                    : = {"reason": "vm not running on<br>this host", "health": "bad", "vm": "down",= "detail": "unknown"}<br>Score             &n= bsp;                : 2400<br>Local= maintenance                  = : False<br>Host timestamp               =       : 1572<br>Extra metadata (valid at timestamp):<br>meta= data_parse_version=3D1<br>metadata_feature_version=3D1<br>timestamp=3D1572 = (Fri Dec 19 10:12:18 2014)<br>host-id=3D2<br>score=3D2400<br>maintenance=3D= False<br>state=3DEngineDown<br><div><br></div><br>--=3D=3D Host 3 status = =3D=3D--<br><div><br></div>Status up-to-date         &n= bsp;        : False<br>Hostname       &n= bsp;                   : 10.0.= 0.92<br>Host ID                 &nb= sp;          : 3<br>Engine status     &n= bsp;                : unknown stale= -data<br>Score                 &nbs= p;            : 2400<br>Local maintenance &nb= sp;                : False<br>Host = timestamp                   &n= bsp; : 987<br>Extra metadata (valid at timestamp):<br>metadata_parse_versio= n=3D1<br>metadata_feature_version=3D1<br>timestamp=3D987 (Fri Dec 19 10:09:= 58 2014)<br>host-id=3D3<br>score=3D2400<br>maintenance=3DFalse<br>state=3DE= ngineDown<br><div><br></div>--<br>And the /var/log/ovirt-hosted-engine-ha/a= gent.log for three ovirt nodes are<br>as follows:<br>--<br>10.0.0.94(hosted= -engine-1)<br>---<br>MainThread::INFO::2014-12-19<br>13:09:33,716::hosted_e= ngine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start= _monitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::2= 014-12-19<br>13:09:33,716::hosted_engine::332::ovirt_hosted_engine_ha.agent= .hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.= 93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:09:44,017::ho= sted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::= (start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::I= NFO::2014-12-19<br>13:09:44,017::hosted_engine::332::ovirt_hosted_engine_ha= .agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 1= 0.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:09:54,3= 03::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEn= gine::(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThr= ead::INFO::2014-12-19<br>13:09:54,303::hosted_engine::332::ovirt_hosted_eng= ine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote = host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:1= 0:04,342::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEng= ine::(consume)<br>Engine vm running on localhost<br>MainThread::INFO::2014-= 12-19<br>13:10:04,617::hosted_engine::327::ovirt_hosted_engine_ha.agent.hos= ted_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (scor= e: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:04,617::hosted_engine::33= 2::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitori= ng)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO:= :2014-12-19<br>13:10:14,657::state_machine::160::ovirt_hosted_engine_ha.age= nt.hosted_engine.HostedEngine::(refresh)<br>Global metadata: {'maintenance'= : False}<br>MainThread::INFO::2014-12-19<br>13:10:14,657::state_machine::16= 5::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Ho= st 10.0.0.93 (id 2): {'extra':<br>'metadata_parse_version=3D1\nmetadata_fea= ture_version=3D1\ntimestamp=3D1448<br>(Fri Dec 19 10:10:14<br>2014)\nhost-i= d=3D2\nscore=3D2400\nmaintenance=3DFalse\nstate=3DEngineDown\n',<br>'hostna= me': '10.0.0.93', 'alive': True, 'host-id': 2, 'engine-status':<br>{'reason= ': 'vm not running on this host', 'health': 'bad', 'vm':<br>'down', 'detail= ': 'unknown'}, 'score': 2400, 'maintenance': False,<br>'host-ts': 1448}<br>= MainThread::INFO::2014-12-19<br>13:10:14,657::state_machine::165::ovirt_hos= ted_engine_ha.agent.hosted_engine.HostedEngine::(refresh)<br>Host 10.0.0.92= (id 3): {'extra':<br>'metadata_parse_version=3D1\nmetadata_feature_version= =3D1\ntimestamp=3D987<br>(Fri Dec 19 10:09:58<br>2014)\nhost-id=3D3\nscore= =3D2400\nmaintenance=3DFalse\nstate=3DEngineDown\n',<br>'hostname': '10.0.0= .92', 'alive': True, 'host-id': 3, 'engine-status':<br>{'reason': 'vm not r= unning on this host', 'health': 'bad', 'vm':<br>'down', 'detail': 'unknown'= }, 'score': 2400, 'maintenance': False,<br>'host-ts': 987}<br>MainThread::I= NFO::2014-12-19<br>13:10:14,658::state_machine::168::ovirt_hosted_engine_ha= .agent.hosted_engine.HostedEngine::(refresh)<br>Local (id 1): {'engine-heal= th': {'health': 'good', 'vm': 'up',<br>'detail': 'up'}, 'bridge': True, 'me= m-free': 1079.0, 'maintenance':<br>False, 'cpu-load': 0.0269, 'gateway': Tr= ue}<br>MainThread::INFO::2014-12-19<br>13:10:14,904::hosted_engine::327::ov= irt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<b= r>Current state EngineUp (score: 2400)<br>MainThread::INFO::2014-12-19<br>1= 3:10:14,904::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine= .HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2, sco= re: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:25,210::hosted_engine::3= 27::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitor= ing)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::2014-12-1= 9<br>13:10:25,210::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_= engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93 (id: = 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:35,499::hosted_eng= ine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_m= onitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INFO::201= 4-12-19<br>13:10:35,499::hosted_engine::332::ovirt_hosted_engine_ha.agent.h= osted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.93= (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:45,784::host= ed_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(s= tart_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThread::INF= O::2014-12-19<br>13:10:45,785::hosted_engine::332::ovirt_hosted_engine_ha.a= gent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.= 0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:10:56,070= ::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngi= ne::(start_monitoring)<br>Current state EngineUp (score: 2400)<br>MainThrea= d::INFO::2014-12-19<br>13:10:56,070::hosted_engine::332::ovirt_hosted_engin= e_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Best remote ho= st 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2014-12-19<br>13:11:= 06,109::states::394::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngin= e::(consume)<br>Engine vm running on localhost<br>MainThread::INFO::2014-12= -19<br>13:11:06,359::hosted_engine::327::ovirt_hosted_engine_ha.agent.hoste= d_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (score:= 2400)<br>MainThread::INFO::2014-12-19<br>13:11:06,359::hosted_engine::332:= :ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring= )<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::INFO::2= 014-12-19<br>13:11:16,658::hosted_engine::327::ovirt_hosted_engine_ha.agent= .hosted_engine.HostedEngine::(start_monitoring)<br>Current state EngineUp (= score: 2400)<br>MainThread::INFO::2014-12-19<br>13:11:16,658::hosted_engine= ::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_moni= toring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThread::I= NFO::2014-12-19<br>13:11:26,991::hosted_engine::327::ovirt_hosted_engine_ha= .agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state Engi= neUp (score: 2400)<br>MainThread::INFO::2014-12-19<br>13:11:26,991::hosted_= engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(star= t_monitoring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>MainThr= ead::INFO::2014-12-19<br>13:11:37,341::hosted_engine::327::ovirt_hosted_eng= ine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current stat= e EngineUp (score: 2400)<br>MainThread::INFO::2014-12-19<br>13:11:37,341::h= osted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:= :(start_monitoring)<br>Best remote host 10.0.0.93 (id: 2, score: 2400)<br>-= ---<br><div><br></div>10.0.0.93 (hosted-engine-2)<br>MainThread::INFO::2014= -12-19<br>10:12:18,339::hosted_engine::327::ovirt_hosted_engine_ha.agent.ho= sted_engine.HostedEngine::(start_monitoring)<br>Current state EngineDown (s= core: 2400)<br>MainThread::INFO::2014-12-19<br>10:12:18,339::hosted_engine:= :332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monit= oring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainThread::IN= FO::2014-12-19<br>10:12:28,651::hosted_engine::327::ovirt_hosted_engine_ha.= agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current state Engin= eDown (score: 2400)<br>MainThread::INFO::2014-12-19<br>10:12:28,652::hosted= _engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(sta= rt_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<br>MainTh= read::INFO::2014-12-19<br>10:12:39,010::hosted_engine::327::ovirt_hosted_en= gine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Current sta= te EngineDown (score: 2400)<br>MainThread::INFO::2014-12-19<br>10:12:39,010= ::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngi= ne::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score: 2400)<b= r>MainThread::INFO::2014-12-19<br>10:12:49,338::hosted_engine::327::ovirt_h= osted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)<br>Cur= rent state EngineDown (score: 2400)<br>MainThread::INFO::2014-12-19<br>10:1= 2:49,338::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_engine.Ho= stedEngine::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1, score:= 2400)<br>MainThread::INFO::2014-12-19<br>10:12:59,642::hosted_engine::327:= :ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring= )<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::2014-12-19= <br>10:12:59,642::hosted_engine::332::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.94 (id: 1= , score: 2400)<br>MainThread::INFO::2014-12-19<br>10:13:10,010::hosted_engi= ne::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_mo= nitoring)<br>Current state EngineDown (score: 2400)<br>MainThread::INFO::20= 14-12-19<br>10:13:10,010::hosted_engine::332::ovirt_hosted_engine_ha.agent.= hosted_engine.HostedEngine::(start_monitoring)<br>Best remote host 10.0.0.9= 4 (id: 1, score: 2400)<br><div><br></div><br>10.0.0.92(hosted-engine-3)<br>= same as 10.0.0.93<br>--<br><div><br></div>-----Original Message-----<br>Fro= m: Simone Tiraboschi [mailto:stirabos@redhat.com]<br>Sent: Friday, December= 19, 2014 12:28 AM<br>To: Yue, Cong<br>Cc: users@ovirt.org<mailto:users@= ovirt.org><mailto:users@ovirt.org><mailto:users@ovirt.org><b= r>Subject: Re: [ovirt-users] VM failover with ovirt3.5<br><div><br></div><b= r><div><br></div>----- Original Message -----<br>From: "Cong Yue" <Cong_= Yue@alliedtelesis.com<mailto:Cong_Yue@alliedtelesis.com><mailto:Co= ng_Yue@alliedtelesis.com><mailto:Cong_Yue@alliedtelesis.com>><b= r>To: users@ovirt.org<mailto:users@ovirt.org><mailto:users@ovirt.o= rg><mailto:users@ovirt.org><br>Sent: Friday, December 19, 2014 2:1= 4:33 AM<br>Subject: [ovirt-users] VM failover with ovirt3.5<br><div><br></d= iv><br><div><br></div>Hi<br><div><br></div><br><div><br></div>In my environ= ment, I have 3 ovirt nodes as one cluster. And on top of<br>host-1, there i= s one vm to host ovirt engine.<br><div><br></div>Also I have one external s= torage for the cluster to use as data domain<br>of engine and data.<br><div= ><br></div>I confirmed live migration works well in my environment.<br><div= ><br></div>But it seems very buggy for VM failover if I try to force to shu= t down<br>one ovirt node. Sometimes the VM in the node which is shutdown ca= n<br>migrate to other host, but it take more than several minutes.<br><div>= <br></div>Sometimes, it can not migrate at all. Sometimes, only when the ho= st is<br>back, the VM is beginning to move.<br><div><br></div>Can you pleas= e check or share the logs under /var/log/ovirt-hosted-engine-ha/<br>?<br><d= iv><br></div>Is there some documentation to explain how VM failover is work= ing? And<br>is there some bugs reported related with this?<br><div><br></di= v>http://www.ovirt.org/Features/Self_Hosted_Engine#Agent_State_Diagram<br><= div><br></div>Thanks in advance,<br><div><br></div>Cong<br><div><br></div><= br><div><br></div><br>This e-mail message is for the sole use of the intend= ed recipient(s)<br>and may contain confidential and privileged information.= Any<br>unauthorized review, use, disclosure or distribution is prohibited.= If<br>you are not the intended recipient, please contact the sender by rep= ly<br>e-mail and destroy all copies of the original message. If you are the= <br>intended recipient, please be advised that the content of this message<= br>is subject to access, review and disclosure by the sender's e-mail Syste= m<br>Administrator.<br><div><br></div>_____________________________________= __________<br>Users mailing list<br>Users@ovirt.org<mailto:Users@ovirt.o= rg><mailto:Users@ovirt.org><mailto:Users@ovirt.org><br>http:= //lists.ovirt.org/mailman/listinfo/users<br><div><br></div>This e-mail mess= age is for the sole use of the intended recipient(s) and may<br>contain con= fidential and privileged information. Any unauthorized review,<br>use, disc= losure or distribution is prohibited. If you are not the intended<br>recipi= ent, please contact the sender by reply e-mail and destroy all copies<br>of= the original message. If you are the intended recipient, please be<br>advi= sed that the content of this message is subject to access, review and<br>di= sclosure by the sender's e-mail System Administrator.<br><div><br></div><br= >This e-mail message is for the sole use of the intended recipient(s) and m= ay contain confidential and privileged information. Any unauthorized review= , use, disclosure or distribution is prohibited. If you are not the intende= d recipient, please contact the sender by reply e-mail and destroy all copi= es of the original message. If you are the intended recipient, please be ad= vised that the content of this message is subject to access, review and dis= closure by the sender's e-mail System Administrator.<br>___________________= ____________________________<br>Users mailing list<br>Users(a)ovirt.org<ma= ilto:Users@ovirt.org><mailto:Users@ovirt.org><mailto:Users@ovir= t.org><br>http://lists.ovirt.org/mailman/listinfo/users<br><div><br></di= v>________________________________<br>This e-mail message is for the sole u= se of the intended recipient(s) and may contain confidential and privileged= information. Any unauthorized review, use, disclosure or distribution is p= rohibited. If you are not the intended recipient, please contact the sender= by reply e-mail and destroy all copies of the original message. If you are= the intended recipient, please be advised that the content of this message= is subject to access, review and disclosure by the sender's e-mail System = Administrator.<br><div><br></div>________________________________<br>This e= -mail message is for the sole use of the intended recipient(s) and may cont= ain confidential and privileged information. Any unauthorized review, use, = disclosure or distribution is prohibited. If you are not the intended recip= ient, please contact the sender by reply e-mail and destroy all copies of t= he original message. If you are the intended recipient, please be advised t= hat the content of this message is subject to access, review and disclosure= by the sender's e-mail System Administrator.<br><div><br></div>___________= _____________________<br>This e-mail message is for the sole use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply e-m= ail and destroy all copies of the original message. If you are the intended= recipient, please be advised that the content of this message is subject t= o access, review and disclosure by the sender's e-mail System Administrator= .<br>-------------- next part --------------<br>An HTML attachment was scru= bbed...<br>URL: <http://lists.ovirt.org/pipermail/users/attachments/2014= 1229/4ec6cc13/attachment.html><br><div><br></div>-----------------------= -------<br><div><br></div>_______________________________________________<b= r>Users mailing list<br>Users(a)ovirt.org<br>http://lists.ovirt.org/mailman/l= istinfo/users<br><div><br></div><br>End of Users Digest, Vol 39, Issue 171<= br>**************************************<br></div><div><br></div></div></b= ody></html> ------=_Part_1882534_1428653136.1419879250032--

9 years, 12 months

1
0
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Users December 2014