<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">Le 19/12/2016 à 08:28, Sahina Bose a
écrit :<br>
</div>
<blockquote
cite="mid:CACjzOvdhyA2-n5MyyVRAgNabzx1NHVRuV-h-hxZSrb3PtYHPVA@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Dec 16, 2016 at 11:00 PM,
Nathanaël Blanchet <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:blanchet@abes.fr"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a></a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class="">
<p><br>
</p>
<br>
<div class="m_-152671267708035808moz-cite-prefix">Le
16/12/2016 à 16:34, Sahina Bose a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Failed to find host
'Host[guadalupe1,7a30c899-<wbr>a317-479a-b07b-244bc2374485]'
in gluster peer list from
'Host[guadalupe1,7a30c899-<wbr>a317-479a-b07b-244bc2374485]'
on attempt 2<br>
It looks the gluster uuid saved in the ovirt
engine db does not match the one returned from
CLI<br>
<br>
</div>
Was this host reinstalled? <br>
</div>
<div>You may need to remove host from engine and
add it again. If that doesn't work you may need
to manually change the uuid value in the
database (gluster_server table)<br>
</div>
</div>
</blockquote>
</span> Removing host did nothing, indeed I had to go to
the gluster_server table to remove any disconnected host
uuid, but it was not enough. Then I had then to remove
the host and reinstall it as a new host.<br>
Thank you, I've been spending a lot of time to solve
this issue.</div>
</blockquote>
<div><br>
</div>
<div>Sorry to hear that you had trouble with this. Could you
explain a bit on how you got into this state?<br>
<br>
</div>
<div>Was it because you re-provisioned one of the gluster
nodes and the gluster UUID was reset (without oVirt being
aware of it?). Would like to either fix/enhance the engine
to handle this if it's a common enough use-case<br>
</div>
</div>
</div>
</div>
</blockquote>
When going to the gluster_server table, I realized that there were
some (disconnected) hosts probed with the gluster network IP. A the
begining, I didn't use the gluster network so my hosts were probed
on the management network and all was fine. When I decided to change
the gluster traffic to a dedicated network (you answered to me about
it there :
<a class="moz-txt-link-freetext" href="https://www.mail-archive.com/users@ovirt.org/msg37742.html">https://www.mail-archive.com/users@ovirt.org/msg37742.html</a>), I
believed that hosts would be probed with the new network IP, but
they didn't. So I manually probed them with the gluster IP, and I
think all my troubles come from there. I reinstalled vdsm, and then
nothing was ok since since this moment.<br>
<blockquote
cite="mid:CACjzOvdhyA2-n5MyyVRAgNabzx1NHVRuV-h-hxZSrb3PtYHPVA@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="h5"><br>
<blockquote type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Dec 16, 2016 at
7:00 PM, Nathanaël Blanchet <span dir="ltr"><<a
moz-do-not-send="true"
class="m_-152671267708035808moz-txt-link-abbreviated"
href="mailto:blanchet@abes.fr"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
extract of the last engine logs, thank you
<div>
<div class="m_-152671267708035808h5"><br>
<br>
<div
class="m_-152671267708035808m_8928738385687730066moz-cite-prefix">Le
16/12/2016 à 14:02, Sahina Bose a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Could you attach the
engine log with this error?<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri,
Dec 16, 2016 at 4:29 PM,
Nathanaël Blanchet <span
dir="ltr"><<a
moz-do-not-send="true"
class="m_-152671267708035808moz-txt-link-abbreviated"
href="mailto:blanchet@abes.fr" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">Hi,<br>
<br>
I used to successfully run a
replica 3 gluster volume, but
since the last 4.0.5 update,
they can't connect each other
with the message : gluster
[gluster peer status <a
moz-do-not-send="true"
href="http://guadalupe1.v100.abes.fr"
rel="noreferrer"
target="_blank">guadalupe1.v100.abes.fr</a>]
command failed on server <a
moz-do-not-send="true"
href="http://guadalupe2.v100.abes.fr"
rel="noreferrer"
target="_blank">guadalupe2.v100.abes.fr</a>.<br>
<br>
So host guadalupe1 can't never
be up.<br>
<br>
When doing gluster peer probe,
they are connected as
expected. I reinstalled vdsm
and gluster, but it is still
the same.<br>
<br>
I found this on guadalupe2
supervdsm.log<br>
<br>
MainProcess|jsonrpc.Executor/6<wbr>::DEBUG::2016-12-16
11:53:21,429::supervdsmServer:<wbr>:99::SuperVdsm.ServerCallback:<wbr>:(wrapper)
return peerStatus with
[{'status': 'CONNECTED',
'hostname': '<a
moz-do-not-send="true"
href="http://10.34.101.56/24"
rel="noreferrer"
target="_blank">10.34.101.56/24</a>',
'uuid':
'c259c09b-8d7c-4b12-8745-67719<wbr>9877583'},
{'status': 'CONNECTED',
'hostname': '<a
moz-do-not-send="true"
href="http://guadalupe3.v100.abes.fr"
rel="noreferrer"
target="_blank">guadalupe3.v100.abes.fr</a>',
'uuid':
'6af67cd3-7931-446d-aaa2-ffea5<wbr>1325adc'},
{'status': 'CONNECTED',
'hostname': '<a
moz-do-not-send="true"
href="http://guadalupe1.v100.abes.fr"
rel="noreferrer"
target="_blank">guadalupe1.v100.abes.fr</a>',
'uuid':
'8eb485cd-31c4-4c3a-a315-3dc6d<wbr>3ddc0c9'}]<br>
MainProcess|jsonrpc.Executor/7<wbr>::DEBUG::2016-12-16
11:53:21,490::supervdsmServer:<wbr>:92::SuperVdsm.ServerCallback:<wbr>:(wrapper)
call peerProbe with () {}<br>
MainProcess|jsonrpc.Executor/7<wbr>::DEBUG::2016-12-16
11:53:21,491::commands::68::ro<wbr>ot::(execCmd)
/usr/bin/taskset --cpu-list
0-63 /usr/sbin/gluster
--mode=script peer probe <a
moz-do-not-send="true"
href="http://guadalupe1.v100.abes.fr"
rel="noreferrer"
target="_blank">guadalupe1.v100.abes.fr</a>
--xml (cwd None)<br>
MainProcess|jsonrpc.Executor/7<wbr>::DEBUG::2016-12-16
11:53:21,570::commands::86::ro<wbr>ot::(execCmd)
SUCCESS: <err> = '';
<rc> = 0<br>
MainProcess|jsonrpc.Executor/7<wbr>::DEBUG::2016-12-16
11:53:21,570::supervdsmServer:<wbr>:99::SuperVdsm.ServerCallback:<wbr>:(wrapper)
return peerProbe with True<br>
<br>
We can see guadalupe2 can see
guadalupe1 but taskset still
executes peer probe to
guadalupe1 with message "Host
<a moz-do-not-send="true"
href="http://guadalupe1.v100.abes.fr"
rel="noreferrer"
target="_blank">guadalupe1.v100.abes.fr</a>
port 24007 already in peer
list"<br>
<br>
How can I say to guadalupe2
stop trying to probe
guadalupe1?<br>
<br>
<br>
-- <br>
Nathanaël Blanchet<br>
<br>
Supervision réseau<br>
Pôle Infrastrutures
Informatiques<br>
227 avenue
Professeur-Jean-Louis-Viala<br>
34193 MONTPELLIER CEDEX 5
<br>
Tél. 33 (0)4 67 54 84 55<br>
Fax 33 (0)4 67 54 84 14<br>
<a moz-do-not-send="true"
href="mailto:blanchet@abes.fr"
target="_blank">blanchet@abes.fr</a><br>
<br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users@ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="m_-152671267708035808m_8928738385687730066moz-signature" cols="72">--
Nathanaël Blanchet
Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
<a moz-do-not-send="true" class="m_-152671267708035808m_8928738385687730066moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr" target="_blank">blanchet@abes.fr</a> </pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
<pre class="m_-152671267708035808moz-signature" cols="72">--
Nathanaël Blanchet
Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
<a moz-do-not-send="true" class="m_-152671267708035808moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr" target="_blank">blanchet@abes.fr</a> </pre>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Nathanaël Blanchet
Supervision réseau
Pôle Infrastrutures Informatiques
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
<a class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a> </pre>
</body>
</html>