--Apple-Mail=_A4BB25DC-3E21-4B00-A859-A2272D61BE0D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
On 18 Nov 2016, at 12:12, Oved Ourfali <oourfali(a)redhat.com>
wrote:
=20
I don't think it is related to version X or Y. It is a race, so might =
be
related to other factors.
=20
likely because json-rpc is initialized after xml-rpc=E2=80=A6.or indeed =
whatever else;-)
either way it needs to be solved. Either by improving the systemd =
service file or mom retry (btw you likely still want to have a retry in =
mom once it starts responding due to delayed vdsm async recovery taking =
potentially minutes)
=20
> On Nov 18, 2016 12:59 PM, "Martin Sivak"
<msivak(a)redhat.com =
<mailto:msivak@redhat.com>> wrote:
> > Are we / can we use systemd socket activation there?
=20
> That actually requires systemd specific code iirc (to
take over the
> standing by socket). I am actually wondering why the xml-rpc in 4.0.4
> was fine and json-rpc in 4.0.6 is too slow.
=20
> Martin
=20
> On Fri, Nov 18, 2016 at 11:53 AM, Anton Marchukov
<amarchuk(a)redhat.com =
<mailto:amarchuk@redhat.com>> wrote:
> Hello All.
>
> Are we / can we use systemd socket activation there?
>
> Anton.
>
> On Fri, Nov 18, 2016 at 11:21 AM, Martin Sivak <msivak(a)redhat.com =
<mailto:msivak@redhat.com>> wrote:
>>
>> What about making vdsm ready to answer connections when it returns =
to
>> systemd instead? I hate workarounds and this always worked
fine.
>>
>> Martin
>>
>> On Fri, Nov 18, 2016 at 11:13 AM, Oved Ourfali <oourfali(a)redhat.com =
<mailto:oourfali@redhat.com>>
>> wrote:
>> > Seems like a race regardless of the protocol.
>> > Should you add a retry?
>> >
>> >
>> > On Nov 18, 2016 11:52 AM, "Martin Sivak" <msivak(a)redhat.com =
<mailto:msivak@redhat.com>> wrote:
>> >>
>> >> Yes, because VDSM is supposed to be up (there is systemd =
dependency).
>> >> This always worked fine with xml-rpc.
>> >>
>> >> Martin
>> >>
>> >> On Fri, Nov 18, 2016 at 10:14 AM, Nir Soffer <nsoffer(a)redhat.com =
<mailto:nsoffer@redhat.com>>
>> >> wrote:
>> >> > On Fri, Nov 18, 2016 at 10:45 AM, Martin Sivak =
<msivak(a)redhat.com <mailto:msivak@redhat.com>>
>> >> > wrote:
>> >> >> This happens because MOM can't connect to VDSM and so it =
quits.
>> >> >
>> >> > So mom try once to connect and if the connection fails it =
quits?
>> >> >
>> >> >> We
>> >> >> discussed it on the mailinglist
>> >> >>
>> >> >>
>> >> >>
>> >> >> =
https://lists.fedoraproject.org/archives/list/vdsm-devel@lists.fedorahoste=
d.org/thread/MZ7UJUWO5KFRDJJDNXX7VIYU5PWSXF62/ =
<
https://lists.fedoraproject.org/archives/list/vdsm-devel@lists.fedorahost=
ed.org/thread/MZ7UJUWO5KFRDJJDNXX7VIYU5PWSXF62/>
>> >> >> =
http://lists.ovirt.org/pipermail/devel/2016-November/014101.html =
<
http://lists.ovirt.org/pipermail/devel/2016-November/014101.html>
>> >> >>
>> >> >> This issue never happened with XML-RPC.
>> >> >>
>> >> >> Shira reported it as
>> >> >>
https://bugzilla.redhat.com/show_bug.cgi?id=3D1393012 =
<
https://bugzilla.redhat.com/show_bug.cgi?id=3D1393012>
>> >> >>
>> >> >> Martin
>> >> >>
>> >> >> On Thu, Nov 17, 2016 at 7:42 PM, Yaniv Kaul
<ykaul(a)redhat.com =
<mailto:ykaul@redhat.com>>
>> >> >> wrote:
>> >> >>> I've recently seen, including now on Master, the
following
>> >> >>> warnings:
>> >> >>> Nov 17 13:33:25 lago-basic-suite-master-host0 systemd[1]:
=
Started
>> >> >>> MOM
>> >> >>> instance configured for VDSM purposes.
>> >> >>> Nov 17 13:33:25 lago-basic-suite-master-host0 systemd[1]:
=
Starting
>> >> >>> MOM
>> >> >>> instance configured for VDSM purposes...
>> >> >>> Nov 17 13:33:35 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available, Policy could not be set.
>> >> >>> Nov 17 13:33:39 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available.
>> >> >>> Nov 17 13:33:39 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available, KSM stats will be missing.
>> >> >>> Nov 17 13:33:55 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available.
>> >> >>> Nov 17 13:33:55 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available, KSM stats will be missing.
>> >> >>> Nov 17 13:34:10 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available.
>> >> >>> Nov 17 13:34:10 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available, KSM stats will be missing.
>> >> >>> Nov 17 13:34:26 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available.
>> >> >>> Nov 17 13:34:26 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available, KSM stats will be missing.
>> >> >>> Nov 17 13:34:42 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available.
>> >> >>> Nov 17 13:34:42 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available, KSM stats will be missing.
>> >> >>> Nov 17 13:34:57 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available.
>> >> >>> Nov 17 13:34:57 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available, KSM stats will be missing.
>> >> >>> Nov 17 13:35:12 lago-basic-suite-master-host0 vdsm[2012]:
=
vdsm MOM
>> >> >>> WARN MOM
>> >> >>> not available.
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Any ideas what this is and why?
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> Devel mailing list
>> >> >>> Devel(a)ovirt.org <mailto:Devel@ovirt.org>
>> >> >>>
http://lists.ovirt.org/mailman/listinfo/devel =
<
http://lists.ovirt.org/mailman/listinfo/devel>
>> >> >>
_______________________________________________
>> >> >> Devel mailing list
>> >> >> Devel(a)ovirt.org <mailto:Devel@ovirt.org>
>> >> >>
http://lists.ovirt.org/mailman/listinfo/devel =
<
http://lists.ovirt.org/mailman/listinfo/devel>
>> >> _______________________________________________
>> >> Devel mailing list
>> >> Devel(a)ovirt.org <mailto:Devel@ovirt.org>
>> >>
http://lists.ovirt.org/mailman/listinfo/devel =
<
http://lists.ovirt.org/mailman/listinfo/devel>
>> >>
>> >>
>> >
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org <mailto:Devel@ovirt.org>
>>
http://lists.ovirt.org/mailman/listinfo/devel =
<
http://lists.ovirt.org/mailman/listinfo/devel>
>
>
>
>
> --
> Anton Marchukov
> Senior Software Engineer - RHEV CI - Red Hat
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_A4BB25DC-3E21-4B00-A859-A2272D61BE0D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On 18 Nov 2016, at 12:12, Oved Ourfali <<a =
href=3D"mailto:oourfali@redhat.com"
class=3D"">oourfali(a)redhat.com</a>&gt;=
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D""><p =
dir=3D"ltr" class=3D"">I don't think it is related to version X
or Y. It =
is a race, so might be related to other factors. =
</p></div></blockquote><div><br
class=3D""></div>likely because json-rpc =
is initialized after xml-rpc=E2=80=A6.or indeed whatever =
else;-)</div><div><br class=3D""></div><div>either
way it needs to be =
solved. Either by improving the systemd service file or mom retry (btw =
you likely still want to have a retry in mom once it starts responding =
due to delayed vdsm async recovery taking potentially =
minutes)</div><div><br class=3D""><blockquote
type=3D"cite" =
class=3D""><div class=3D"">
<div class=3D"gmail_extra"><br class=3D""><div
class=3D"gmail_quote">On =
Nov 18, 2016 12:59 PM, "Martin Sivak" <<a =
href=3D"mailto:msivak@redhat.com"
class=3D"">msivak(a)redhat.com</a>&gt; =
wrote:<br type=3D"attribution" class=3D""><blockquote =
class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc =
solid;padding-left:1ex">> Are we / can we use systemd socket =
activation there?<br class=3D"">
<br class=3D"">
That actually requires systemd specific code iirc (to take over the<br =
class=3D"">
standing by socket). I am actually wondering why the xml-rpc in 4.0.4<br =
class=3D"">
was fine and json-rpc in 4.0.6 is too slow.<br class=3D"">
<br class=3D"">
Martin<br class=3D"">
<br class=3D"">
On Fri, Nov 18, 2016 at 11:53 AM, Anton Marchukov <<a =
href=3D"mailto:amarchuk@redhat.com"
class=3D"">amarchuk(a)redhat.com</a>&gt;=
wrote:<br class=3D"">
> Hello All.<br class=3D"">
><br class=3D"">
> Are we / can we use systemd socket activation there?<br
class=3D"">
><br class=3D"">
> Anton.<br class=3D"">
><br class=3D"">
> On Fri, Nov 18, 2016 at 11:21 AM, Martin Sivak <<a =
href=3D"mailto:msivak@redhat.com"
class=3D"">msivak(a)redhat.com</a>&gt; =
wrote:<br class=3D"">
>><br class=3D"">
>> What about making vdsm ready to answer connections when it =
returns to<br class=3D"">
>> systemd instead? I hate workarounds and this always worked =
fine.<br class=3D"">
>><br class=3D"">
>> Martin<br class=3D"">
>><br class=3D"">
>> On Fri, Nov 18, 2016 at 11:13 AM, Oved Ourfali <<a =
href=3D"mailto:oourfali@redhat.com" =
class=3D"">oourfali(a)redhat.com</a>&gt;<br
class=3D"">
>> wrote:<br class=3D"">
>> > Seems like a race regardless of the protocol.<br
class=3D"">=
>> > Should you add a retry?<br class=3D"">
>> ><br class=3D"">
>> ><br class=3D"">
>> > On Nov 18, 2016 11:52 AM, "Martin Sivak" <<a
=
href=3D"mailto:msivak@redhat.com"
class=3D"">msivak(a)redhat.com</a>&gt; =
wrote:<br class=3D"">
>> >><br class=3D"">
>> >> Yes, because VDSM is supposed to be up (there is =
systemd dependency).<br class=3D"">
>> >> This always worked fine with xml-rpc.<br
class=3D"">
>> >><br class=3D"">
>> >> Martin<br class=3D"">
>> >><br class=3D"">
>> >> On Fri, Nov 18, 2016 at 10:14 AM, Nir Soffer
<<a =
href=3D"mailto:nsoffer@redhat.com" =
class=3D"">nsoffer(a)redhat.com</a>&gt;<br
class=3D"">
>> >> wrote:<br class=3D"">
>> >> > On Fri, Nov 18, 2016 at 10:45 AM, Martin Sivak
=
<<a href=3D"mailto:msivak@redhat.com" =
class=3D"">msivak(a)redhat.com</a>&gt;<br
class=3D"">
>> >> > wrote:<br class=3D"">
>> >> >> This happens because MOM can't
connect to =
VDSM and so it quits.<br class=3D"">
>> >> ><br class=3D"">
>> >> > So mom try once to connect and if the
connection =
fails it quits?<br class=3D"">
>> >> ><br class=3D"">
>> >> >> We<br class=3D"">
>> >> >> discussed it on the mailinglist<br
class=3D"">
>> >> >><br class=3D"">
>> >> >><br class=3D"">
>> >> >><br class=3D"">
>> >> >> <a =
href=3D"https://lists.fedoraproject.org/archives/list/vdsm-devel@lis...
orahosted.org/thread/MZ7UJUWO5KFRDJJDNXX7VIYU5PWSXF62/"
rel=3D"noreferrer"=
target=3D"_blank" class=3D"">https://lists.fedoraproject.<wbr
=
class=3D"">org/archives/list/vdsm-devel@<wbr =
class=3D"">lists.fedorahosted.org/thread/<wbr =
class=3D"">MZ7UJUWO5KFRDJJDNXX7VIYU5PWSXF<wbr
class=3D"">62/</a><br =
class=3D"">
>> >> >> <a =
href=3D"http://lists.ovirt.org/pipermail/devel/2016-November/014101.... =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">pipermail/devel/2016-November/<wbr =
class=3D"">014101.html</a><br class=3D"">
>> >> >><br class=3D"">
>> >> >> This issue never happened with
XML-RPC.<br =
class=3D"">
>> >> >><br class=3D"">
>> >> >> Shira reported it as<br
class=3D"">
>> >> >> <a =
href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1393012" =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">https://bugzilla.redhat.com/<wbr =
class=3D"">show_bug.cgi?id=3D1393012</a><br
class=3D"">
>> >> >><br class=3D"">
>> >> >> Martin<br class=3D"">
>> >> >><br class=3D"">
>> >> >> On Thu, Nov 17, 2016 at 7:42 PM, Yaniv
Kaul =
<<a href=3D"mailto:ykaul@redhat.com" =
class=3D"">ykaul(a)redhat.com</a>&gt;<br class=3D"">
>> >> >> wrote:<br class=3D"">
>> >> >>> I've recently seen,
including now on =
Master, the following<br class=3D"">
>> >> >>> warnings:<br
class=3D"">
>> >> >>> Nov 17 13:33:25 =
lago-basic-suite-master-host0 systemd[1]: Started<br class=3D"">
>> >> >>> MOM<br
class=3D"">
>> >> >>> instance configured for VDSM
purposes.<br =
class=3D"">
>> >> >>> Nov 17 13:33:25 =
lago-basic-suite-master-host0 systemd[1]: Starting<br class=3D"">
>> >> >>> MOM<br
class=3D"">
>> >> >>> instance configured for VDSM =
purposes...<br class=3D"">
>> >> >>> Nov 17 13:33:35 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available, Policy could not
be =
set.<br class=3D"">
>> >> >>> Nov 17 13:33:39 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available.<br
class=3D"">
>> >> >>> Nov 17 13:33:39 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available, KSM stats will
be =
missing.<br class=3D"">
>> >> >>> Nov 17 13:33:55 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available.<br
class=3D"">
>> >> >>> Nov 17 13:33:55 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available, KSM stats will
be =
missing.<br class=3D"">
>> >> >>> Nov 17 13:34:10 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available.<br
class=3D"">
>> >> >>> Nov 17 13:34:10 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available, KSM stats will
be =
missing.<br class=3D"">
>> >> >>> Nov 17 13:34:26 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available.<br
class=3D"">
>> >> >>> Nov 17 13:34:26 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available, KSM stats will
be =
missing.<br class=3D"">
>> >> >>> Nov 17 13:34:42 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available.<br
class=3D"">
>> >> >>> Nov 17 13:34:42 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available, KSM stats will
be =
missing.<br class=3D"">
>> >> >>> Nov 17 13:34:57 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available.<br
class=3D"">
>> >> >>> Nov 17 13:34:57 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available, KSM stats will
be =
missing.<br class=3D"">
>> >> >>> Nov 17 13:35:12 =
lago-basic-suite-master-host0 vdsm[2012]: vdsm MOM<br class=3D"">
>> >> >>> WARN MOM<br
class=3D"">
>> >> >>> not available.<br
class=3D"">
>> >> >>><br class=3D"">
>> >> >>><br class=3D"">
>> >> >>><br class=3D"">
>> >> >>> Any ideas what this is and
why?<br =
class=3D"">
>> >> >>><br class=3D"">
>> >> >>>
______________________________<wbr =
class=3D"">_________________<br class=3D"">
>> >> >>> Devel mailing list<br
class=3D"">
>> >> >>> <a
href=3D"mailto:Devel@ovirt.org" =
class=3D"">Devel(a)ovirt.org</a><br class=3D"">
>> >> >>> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel"
rel=3D"noreferrer" =
target=3D"_blank"
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/devel</a><br class=3D"">
>> >> >> ______________________________<wbr
=
class=3D"">_________________<br class=3D"">
>> >> >> Devel mailing list<br
class=3D"">
>> >> >> <a
href=3D"mailto:Devel@ovirt.org" =
class=3D"">Devel(a)ovirt.org</a><br class=3D"">
>> >> >> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel"
rel=3D"noreferrer" =
target=3D"_blank"
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/devel</a><br class=3D"">
>> >> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>> >> Devel mailing list<br class=3D"">
>> >> <a href=3D"mailto:Devel@ovirt.org" =
class=3D"">Devel(a)ovirt.org</a><br class=3D"">
>> >> <a =
href=3D"http://lists.ovirt.org/mailman/listinfo/devel"
rel=3D"noreferrer" =
target=3D"_blank"
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/devel</a><br class=3D"">
>> >><br class=3D"">
>> >><br class=3D"">
>> ><br class=3D"">
>> ______________________________<wbr =
class=3D"">_________________<br class=3D"">
>> Devel mailing list<br class=3D"">
>> <a href=3D"mailto:Devel@ovirt.org" =
class=3D"">Devel(a)ovirt.org</a><br class=3D"">
>> <a
href=3D"http://lists.ovirt.org/mailman/listinfo/devel" =
rel=3D"noreferrer" target=3D"_blank" =
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/devel</a><br class=3D"">
><br class=3D"">
><br class=3D"">
><br class=3D"">
><br class=3D"">
> --<br class=3D"">
> Anton Marchukov<br class=3D"">
> Senior Software Engineer - RHEV CI - Red Hat<br class=3D"">
><br class=3D"">
</blockquote></div></div>
_______________________________________________<br class=3D"">Devel =
mailing list<br class=3D""><a href=3D"mailto:Devel@ovirt.org"
=
class=3D"">Devel(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/devel<...
</div><br class=3D""></body></html>=
--Apple-Mail=_A4BB25DC-3E21-4B00-A859-A2272D61BE0D--