9:22 a.m.
------=_Part_234528_2091500974.1414657372046
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi Guys,
I need help with my ovirt Hosted-Engine HA setup. I am running on 2 ovirt hosts and 2
gluster nodes with replicated volumes. i already have VMs running on my hosts and they can
migrate normally once i for example power off the host that they are running on. the
problem is that the engine can't migrate once i switch off the host that hosts the
engine.
oVirt 3.4.3-1.el6
KVM 0.12.1.2 - 2.415.el6_5.10
LIBVIRT libvirt-0.10.2-29.el6_5.9
VDSM vdsm-4.14.17-0.el6
right now, i have this result from hosted-engine --vm-status.
BQ_BEGIN
File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
exec code in run_globals
File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py",
line 111, in <module>
if not status_checker.print_status():
File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py",
line 58, in print_status
all_host_stats = ha_cli.get_all_host_stats()
File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 137, in get_all_host_stats
return self.get_all_stats(self.StatModes.HOST)
File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 86, in get_all_stats
constants.SERVICE_TYPE)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 171, in get_stats_from_storage
result = self._checked_communicate(request)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 199, in _checked_communicate
.format(message or response))
ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: <type
'exceptions.OSError'>
BQ_END
restarting ha-broker and ha-agent normalizes the status but eventually it would become
"false" and then return to the result above. hope you guys could help me with
this.
Thanks,
Jaicel
------=_Part_234528_2091500974.1414657372046
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"font-family: arial, helvetica, sans-serif;
font-s=
ize: 10pt; color: #000000"><div>Hi
Guys,<br></div><div><br data-mce-bogus=
=3D"1"></div><div>I need help with my ovirt Hosted-Engine HA
setup. I am ru=
nning on 2 ovirt hosts and 2 gluster nodes with replicated volumes. i alrea=
dy have VMs running on my hosts and they can migrate normally once i for ex=
ample power off the host that they are running on. the problem is that the =
engine can't migrate once i switch off the host that hosts the engine. <br =
data-mce-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><blockquote>=
<div><span data-mce-style=3D"color: #111111; font-family: Verdana;
font-siz=
e: 10pt; font-style: normal; font-variant: normal; font-weight: normal; let=
ter-spacing: normal; line-height: 24px; text-align: left; text-indent: 0px;=
text-transform: none; white-space: normal; word-spacing: 0px; display: inl=
ine ! important; float: none;" style=3D"color: rgb(17, 17, 17); font-family=
: Verdana; font-size: 10pt; font-style: normal; font-variant: normal; font-=
weight: normal; letter-spacing: normal; line-height: 24px; text-align: left=
; text-indent: 0px; text-transform: none; white-space: normal; word-spacing=
: 0px; display: inline ! important; float: none;">oVirt
=
3.4.3-1.el6</span></div><div>KVM
&nbs=
p; 0.12.1.2 -
2.415.el6_5.10</div><div>LIBVIRT libvi=
rt-0.10.2-29.el6_5.9<br
data-mce-bogus=3D"1"></div><div>VDSM &nb=
sp; vdsm-4.14.17-0.el6<br
data-mce-bogus=3D"1"></div></blockquote><d=
iv><br data-mce-bogus=3D"1"></div><div>right now, i have
this result from h=
osted-engine --vm-status.<br
data-mce-bogus=3D"1"></div><div><br data-mce-b=
ogus=3D"1"></div><blockquote><div> File
"/usr/lib64/python2.6/runpy.p=
y", line 122, in _run_module_as_main<br>
"__main__", fnam=
e, loader, pkg_name)<br> File "/usr/lib64/python2.6/runpy.py",
line 3=
4, in _run_code<br> exec code in
run_globals<br> Fi=
le "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py=
", line 111, in <module><br> if not
status_checker.=
print_status():<br> File
"/usr/lib/python2.6/site-packages/ovirt_host=
ed_engine_setup/vm_status.py", line 58, in
print_status<br> &nbs=
p; all_host_stats =3D ha_cli.get_all_host_stats()<br> File
"/usr/lib/=
python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 137,=
in get_all_host_stats<br> return
self.get_all_stats(self=
.StatModes.HOST)<br> File
"/usr/lib/python2.6/site-packages/ovirt_hos=
ted_engine_ha/client/client.py", line 86, in
get_all_stats<br> &=
nbsp; constants.SERVICE_TYPE)<br> File
"/usr/lib/python2.6/site-packa=
ges/ovirt_hosted_engine_ha/lib/brokerlink.py", line 171, in get_stats_from_=
storage<br> result =3D
self._checked_communicate(request)=
<br> File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/li=
b/brokerlink.py", line 199, in
_checked_communicate<br> .=
format(message or response))<br>ovirt_hosted_engine_ha.lib.exceptions.Reque=
stError: Request failed: <type
'exceptions.OSError'></div></blockquot=
e><div><br data-mce-bogus=3D"1"></div><div>restarting
ha-broker and ha-agen=
t normalizes the status but eventually it would become "false" and then ret=
urn to the result above. hope you guys could help me with this.<br data-mce=
-bogus=3D"1"></div><div><br
data-mce-bogus=3D"1"></div><div data-marker=3D"=
__SIG_PRE__">Thanks,<br>Jaicel</div></div></body></html>
------=_Part_234528_2091500974.1414657372046--
2:15 p.m.
On 10/30/2014 09:22 AM, Jaicel R. Sabonsolin wrote:
Hi Guys,
I need help with my ovirt Hosted-Engine HA setup. I am running on 2
ovirt hosts and 2 gluster nodes with replicated volumes. i already have
VMs running on my hosts and they can migrate normally once i for example
power off the host that they are running on. the problem is that the
engine can't migrate once i switch off the host that hosts the engine.
oVirt 3.4.3-1.el6
KVM 0.12.1.2 - 2.415.el6_5.10
LIBVIRT libvirt-0.10.2-29.el6_5.9
VDSM vdsm-4.14.17-0.el6
right now, i have this result from hosted-engine --vm-status.
File "/usr/lib64/python2.6/runpy.py", line 122, in
_run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
exec code in run_globals
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py",
line 111, in <module>
if not status_checker.print_status():
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py",
line 58, in print_status
all_host_stats = ha_cli.get_all_host_stats()
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 137, in get_all_host_stats
return self.get_all_stats(self.StatModes.HOST)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
line 86, in get_all_stats
constants.SERVICE_TYPE)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 171, in get_stats_from_storage
result = self._checked_communicate(request)
File
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 199, in _checked_communicate
.format(message or response))
ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed:
<type 'exceptions.OSError'>
restarting ha-broker and ha-agent normalizes the status but eventually
it would become "false" and then return to the result above. hope you
guys could help me with this.
Hi Jaicel,
please attach agent.log and broker.log from the host where you trying to
run hosted-engine --vm-status. I have a feeling that you ran into a
known problem on gluster - stalled file descriptor, in that case the
only known solution at this time is to restart the broker & agent as you
have already found out.
Regards,
Jirka
Thanks,
Jaicel
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:37 p.m.
On 10/30/2014 06:45 PM, Jiri Moskovcak wrote:
On 10/30/2014 09:22 AM, Jaicel R. Sabonsolin wrote:
> Hi Guys,
>
> I need help with my ovirt Hosted-Engine HA setup. I am running on 2
> ovirt hosts and 2 gluster nodes with replicated volumes. i already have
> VMs running on my hosts and they can migrate normally once i for example
> power off the host that they are running on. the problem is that the
> engine can't migrate once i switch off the host that hosts the engine.
>
> oVirt 3.4.3-1.el6
> KVM 0.12.1.2 - 2.415.el6_5.10
> LIBVIRT libvirt-0.10.2-29.el6_5.9
> VDSM vdsm-4.14.17-0.el6
>
>
> right now, i have this result from hosted-engine --vm-status.
>
> File "/usr/lib64/python2.6/runpy.py", line 122, in
> _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
> exec code in run_globals
> File
>
> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py",
>
> line 111, in <module>
> if not status_checker.print_status():
> File
>
> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py",
>
> line 58, in print_status
> all_host_stats = ha_cli.get_all_host_stats()
> File
>
>
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
>
> line 137, in get_all_host_stats
> return self.get_all_stats(self.StatModes.HOST)
> File
>
>
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
>
> line 86, in get_all_stats
> constants.SERVICE_TYPE)
> File
>
>
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>
> line 171, in get_stats_from_storage
> result = self._checked_communicate(request)
> File
>
>
"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>
> line 199, in _checked_communicate
> .format(message or response))
> ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed:
> <type 'exceptions.OSError'>
>
>
> restarting ha-broker and ha-agent normalizes the status but eventually
> it would become "false" and then return to the result above. hope you
> guys could help me with this.
>
Hi Jaicel,
please attach agent.log and broker.log from the host where you trying to
run hosted-engine --vm-status. I have a feeling that you ran into a
known problem on gluster - stalled file descriptor, in that case the
only known solution at this time is to restart the broker & agent as you
have already found out.
Adding Niels and gluster-devel to troubleshoot from Gluster NFS perspective.
Thanks,
Vijay
9:11 p.m.
On Thu, Oct 30, 2014 at 09:07:24PM +0530, Vijay Bellur wrote:
On 10/30/2014 06:45 PM, Jiri Moskovcak wrote:
>On 10/30/2014 09:22 AM, Jaicel R. Sabonsolin wrote:
>>Hi Guys,
>>
>>I need help with my ovirt Hosted-Engine HA setup. I am running on 2
>>ovirt hosts and 2 gluster nodes with replicated volumes. i already have
>>VMs running on my hosts and they can migrate normally once i for example
>>power off the host that they are running on. the problem is that the
>>engine can't migrate once i switch off the host that hosts the engine.
>>
>> oVirt 3.4.3-1.el6
>> KVM 0.12.1.2 - 2.415.el6_5.10
>> LIBVIRT libvirt-0.10.2-29.el6_5.9
>> VDSM vdsm-4.14.17-0.el6
>>
>>
>>right now, i have this result from hosted-engine --vm-status.
>>
>> File "/usr/lib64/python2.6/runpy.py", line 122, in
>> _run_module_as_main
>> "__main__", fname, loader, pkg_name)
>> File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
>> exec code in run_globals
>> File
>>
>>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py",
>>
>> line 111, in <module>
>> if not status_checker.print_status():
>> File
>>
>>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_setup/vm_status.py",
>>
>> line 58, in print_status
>> all_host_stats = ha_cli.get_all_host_stats()
>> File
>>
>>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
>>
>> line 137, in get_all_host_stats
>> return self.get_all_stats(self.StatModes.HOST)
>> File
>>
>>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/client/client.py",
>>
>> line 86, in get_all_stats
>> constants.SERVICE_TYPE)
>> File
>>
>>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>
>> line 171, in get_stats_from_storage
>> result = self._checked_communicate(request)
>> File
>>
>>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>
>> line 199, in _checked_communicate
>> .format(message or response))
>> ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed:
>> <type 'exceptions.OSError'>
>>
>>
>>restarting ha-broker and ha-agent normalizes the status but eventually
>>it would become "false" and then return to the result above. hope you
>>guys could help me with this.
>>
>
>Hi Jaicel,
>please attach agent.log and broker.log from the host where you trying to
>run hosted-engine --vm-status. I have a feeling that you ran into a
>known problem on gluster - stalled file descriptor, in that case the
>only known solution at this time is to restart the broker & agent as you
>have already found out.
>
Adding Niels and gluster-devel to troubleshoot from Gluster NFS perspective.
I'd welcome any details on this "stalled file descriptor" problem. Is
there a bug filed with some details like logs, sysrq-t and maybe even
tcpdumps? If there is an easy way to reproduce this behaviour, I can
surely look into it and hopefully come up with some advise or fix.
Thanks,
Niels
3469
days inactive
3469
days old
3 comments
4 participants
participants (4)
-
Jaicel R. Sabonsolin -
Jiri Moskovcak -
Niels de Vos -
Vijay Bellur