<div dir="ltr">Adding Evgheni.</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 19, 2017 at 10:01 AM, Nadav Goldin <span dir="ltr"><<a href="mailto:ngoldin@redhat.com" target="_blank">ngoldin@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Milan, sorry for missing this.<br>
<br>
In short, it looks like a libvirt/qemu error, I guess it lays<br>
somewhere in the nested environment the Jenkins slave runs at. I was<br>
able to extract the libvirt log from this specific run, but there is<br>
nothing useful there, except that there was no proper termination.<br>
>From reading here[1] it might be related to a load on the hypervisor,<br>
and the timeout configured for libvirt to wait for qemu. Unfortunately<br>
looking at the this[2] thread, it seems that a patch to configure the<br>
timeout never got into libvirt, which leaves us with a default of 30<br>
seconds, and that might not be enough in our nested environment. I<br>
presume that if the hypervisor which the Jenkins slave runs is highly<br>
loaded, then when we try to start the vdsm_functional_tests_lago VM,<br>
it might take more than 30 seconds for qemu to respond.<br>
<br>
Another indication of this "hypothesis" is that I never seen this<br>
error on OST - which uses bare-metal slaves.<br>
<br>
Evgheni, do we have the load monitoring on the hypervisor that runs<br>
<a href="http://vm0065.workers-phx.ovirt.org" rel="noreferrer" target="_blank">vm0065.workers-phx.ovirt.org</a>? Not sure if we added that eventually.<br>
<br>
<br>
[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=987088" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=987088</a><br>
[2] <a href="https://www.redhat.com/archives/libvir-list/2014-January/msg00410.html" rel="noreferrer" target="_blank">https://www.redhat.com/<wbr>archives/libvir-list/2014-<wbr>January/msg00410.html</a><br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Apr 10, 2017 at 10:56 AM, Milan Zamazal <<a href="mailto:mzamazal@redhat.com">mzamazal@redhat.com</a>> wrote:<br>
> Hi,<br>
><br>
> after my Vdsm patch <a href="https://gerrit.ovirt.org/75329" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/75329</a> in ovirt-4.1 branch<br>
> had been merged, Jenkins check-merged job<br>
> <a href="http://jenkins.ovirt.org/job/vdsm_4.1_check-merged-el7-x86_64/173/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>vdsm_4.1_check-merged-el7-x86_<wbr>64/173/</a><br>
> failed with the following error:<br>
><br>
> 07:01:21 @ Start specified VMs:<br>
> 07:01:21 # Start nets:<br>
> 07:01:21 * Create network vdsm_functional_tests_lago:<br>
> 07:01:27 * Create network vdsm_functional_tests_lago: Success (in 0:00:05)<br>
> 07:01:27 # Start nets: Success (in 0:00:05)<br>
> 07:01:27 # Start vms:<br>
> 07:01:27 * Starting VM vdsm_functional_tests_host-<wbr>el7:<br>
> 07:02:07 libvirt: QEMU Driver error : monitor socket did not show up: No such file or directory<br>
> 07:02:07 * Starting VM vdsm_functional_tests_host-<wbr>el7: ERROR (in 0:00:40)<br>
> 07:02:07 # Start vms: ERROR (in 0:00:40)<br>
> 07:02:07 # Destroy network vdsm_functional_tests_lago:<br>
> 07:02:07 # Destroy network vdsm_functional_tests_lago: ERROR (in 0:00:00)<br>
> 07:02:07 @ Start specified VMs: ERROR (in 0:00:46)<br>
> 07:02:07 Error occured, aborting<br>
> 07:02:07 Traceback (most recent call last):<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/cmd.py", line 936, in main<br>
> 07:02:07 cli_plugins[args.verb].do_run(<wbr>args)<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/cli.py", line 184, in do_run<br>
> 07:02:07 self._do_run(**vars(args))<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 495, in wrapper<br>
> 07:02:07 return func(*args, **kwargs)<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/utils.py", line 506, in wrapper<br>
> 07:02:07 return func(*args, prefix=prefix, **kwargs)<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/cmd.py", line 264, in do_start<br>
> 07:02:07 prefix.start(vm_names=vm_<wbr>names)<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/prefix.py", line 1033, in start<br>
> 07:02:07 self.virt_env.start(vm_names=<wbr>vm_names)<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/virt.py", line 331, in start<br>
> 07:02:07 vm.start()<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/plugins/vm.py", line 299, in start<br>
> 07:02:07 return self.provider.start(*args, **kwargs)<br>
> 07:02:07 File "/usr/lib/python2.7/site-<wbr>packages/lago/vm.py", line 106, in start<br>
> 07:02:07 dom = self.libvirt_con.createXML(<wbr>self._libvirt_xml())<br>
> 07:02:07 File "/usr/lib64/python2.7/site-<wbr>packages/libvirt.py", line 3782, in createXML<br>
> 07:02:07 if ret is None:raise libvirtError('<wbr>virDomainCreateXML() failed', conn=self)<br>
> 07:02:07 libvirtError: monitor socket did not show up: No such file or directory<br>
> 07:02:07 Took 210 seconds<br>
><br>
> The error is apparently unrelated to my patch since: 1. my patch should<br>
> have nothing to do with VM start; 2. Jenkins has run successfully on the<br>
> following patch (<a href="https://gerrit.ovirt.org/75321" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/<wbr>75321</a>). FWIW, the preceding<br>
> patch (<a href="https://gerrit.ovirt.org/75038" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/<wbr>75038</a>) has run successfully too.<br>
><br>
> Do you know what's wrong?<br>
><br>
> Thanks,<br>
> Milan<br>
> ______________________________<wbr>_________________<br>
> Infra mailing list<br>
> <a href="mailto:Infra@ovirt.org">Infra@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/infra</a><br>
______________________________<wbr>_________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org">Infra@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/infra</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">ASSOCIATE MANAGER</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat EMEA</a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></td></tr></tbody></table></div><div>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div></div></div>
</div>