<div dir="ltr">I have now enqueued another run of OST for the next VDSM patch in line (that was merged around the time of the first failure) <a href="https://gerrit.ovirt.org/#/c/75299/2" target="_blank">https://gerrit.ovirt.<wbr>org/#/c/75299/2</a> to see if it was the culprit.<div>Not sure why we allow only one (<b>globally for all developers) </b>concurrent run of the manual OST job... I've created a jira issue for this <a href="https://ovirt-jira.atlassian.net/browse/OST-61">https://ovirt-jira.atlassian.net/browse/OST-61</a></div><div><br></div><div>Martin</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 10, 2017 at 3:29 PM, Martin Betak <span dir="ltr"><<a href="mailto:mbetak@redhat.com" target="_blank">mbetak@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Actually it appears that this failure is not engine related (tried reverting my patches from saturday but still received similar errors with lago in the <span style="font-size:17.6px">[ add_secondary_storag</span><span style="font-size:17.6px">e_domains] test</span>).<div>But when running the current engine master with VDSM cutoff before the time of failure, it passed <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_manual/229/" target="_blank">http://jenkins.ovirt.or<wbr>g/job/ovirt-system-tests_manua<wbr>l/229/</a></div><div><br></div><div>This was the VDSM commit that *worked* </div><table class="gmail-m_4354741040091692675m_2155546378191710501gmail-pane" style="box-sizing:border-box;font-family:helvetica,arial,sans-serif;font-size:14px;line-height:1.4em;color:rgb(51,51,51);width:1355.45px;border-collapse:collapse;border:none"><tbody style="box-sizing:border-box"><tr class="gmail-m_4354741040091692675m_2155546378191710501gmail-pane" style="box-sizing:border-box"><td colspan="2" class="gmail-m_4354741040091692675m_2155546378191710501gmail-changeset" style="box-sizing:border-box;font-family:helvetica,arial,sans-serif;line-height:1.4em;vertical-align:middle;padding:4px 4px 3px"><a name="m_4354741040091692675_m_2155546378191710501_5c8aff6177bdaa81ee11c20d417c9bb10e651fb8" style="box-sizing:border-box"></a><div class="gmail-m_4354741040091692675m_2155546378191710501gmail-changeset-message" style="box-sizing:border-box;border:1px solid rgb(204,204,187);background:rgb(238,238,221);padding:4px"><b style="box-sizing:border-box">Commit 5c8aff6177bdaa81ee11c20d417c9b<wbr>b10e651fb8 by <a href="http://jenkins.ovirt.org/user/fromani@redhat.com/" style="box-sizing:border-box;color:rgb(92,53,102)" target="_blank">Francesco Romani</a></b></div></td></tr></tbody></table></div><div class="gmail-m_4354741040091692675HOEnZb"><div class="gmail-m_4354741040091692675h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 10, 2017 at 11:08 AM, Michal Skrivanek <span dir="ltr"><<a href="mailto:michal.skrivanek@redhat.com" target="_blank">michal.skrivanek@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><br>
> On 9 Apr 2017, at 09:42, Barak Korren <<a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>> wrote:<br>
><br>
> On 9 April 2017 at 10:39, Yaniv Kaul <<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>> wrote:<br>
>> 1. The error:<br>
>> 2017-04-08 14:12:11,376-04 ERROR<br>
>> [org.ovirt.vdsm.jsonrpc.client<wbr>.JsonRpcClient] (ResponseWorker) [] Not able<br>
>> to update response for "fc0cef3f-d8fa-4074-b315-e36cc<wbr>2f63fa1"<br>
>><br>
>> Is still seen, not sure why.<br>
>><br>
>> 2. MOM is not available (unrelated, but still a bug)<br>
>><br>
>> 3. New error I have not seen in the past on host1<br>
>> (<a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6257/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host1/_var_log/vdsm/vdsm.log" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_m<wbr>aster/6257/artifact/exported-a<wbr>rtifacts/basic-suit-master-el7<wbr>/test_logs/basic-suite-master/<wbr>post-002_bootstrap.py/lago-bas<wbr>ic-suite-master-host1/_var_log<wbr>/vdsm/vdsm.log</a><br>
>> ) :<br>
>><br>
>> 2017-04-08 14:12:22,208-0400 WARN (jsonrpc/6) [storage.LVM] lvm pvs failed:<br>
>> 5 [] [' Failed to find physical volume<br>
>> "/dev/mapper/360014050eacbb3c8<wbr>b21428fb6683f074".'] (lvm:325)<br>
>><br>
>> 4. New warning:<br>
>> 2017-04-08 14:13:17,327-0400 WARN (check/loop) [storage.asyncutils] Call<br>
>> <bound method DirectioChecker._check of <DirectioChecker<br>
>> /dev/0eb0f05c-0507-4f02-ba13-b<wbr>8d865538d7a/metadata running<br>
>> next_check=4295183.36 at 0x2fb9190>> delayed by 1.43 seconds<br>
>> (asyncutils:138)<br>
>><br>
>><br>
>> Do we know when it began?<br>
><br>
> Yes. I linked to the patch that seems to have started this.<br>
<br>
</span>This should be easy to revert<br>
Working on it<br>
Then we’ll see what exactly is the culprit<br>
<span class="gmail-m_4354741040091692675m_2155546378191710501im gmail-m_4354741040091692675m_2155546378191710501HOEnZb"><br>
><br>
><br>
><br>
> --<br>
> Barak Korren<br>
> <a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a><br>
> RHCE, RHCi, RHV-DevOps Team<br>
> <a href="https://ifireball.wordpress.com/" rel="noreferrer" target="_blank">https://ifireball.wordpress.co<wbr>m/</a><br>
</span><div class="gmail-m_4354741040091692675m_2155546378191710501HOEnZb"><div class="gmail-m_4354741040091692675m_2155546378191710501h5">> ______________________________<wbr>_________________<br>
> Devel mailing list<br>
> <a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
><br>
><br>
<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a></div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>