<div dir="ltr">Renaming title and adding devel.<br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 20, 2016 at 2:36 PM, Piotr Kliczewski <span dir="ltr"><<a href="mailto:pkliczew@redhat.com" target="_blank">pkliczew@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>The last failure seems to be storage related.<br><br></div><div>@Nir please take a look.<br></div><div><br></div>Here is engine side error:<br><br>2016-11-20 05:54:59,605 DEBUG [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>ConnectStoragePoolVDSCommand] (default task-5) [59fc0074] Exception: org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.<wbr>IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=1ca141f1-b64d-4a52-<wbr>8861-05c7de2a72b2, msdUUID=7d4bf750-4fb8-463f-<wbr>bbb0-92156c47306e'<br><br></div>and here is vdsm:<br><br>jsonrpc.Executor/5::ERROR::<wbr>2016-11-20 05:54:56,331::multipath::95::<wbr>Storage.Multipath::(resize_<wbr>devices) Could not resize device 360014052749733c7b8248628637b9<wbr>90f<br>Traceback (most recent call last):<br> File "/usr/share/vdsm/storage/<wbr>multipath.py", line 93, in resize_devices<br> _resize_if_needed(guid)<br> File "/usr/share/vdsm/storage/<wbr>multipath.py", line 101, in _resize_if_needed<br> for slave in devicemapper.getSlaves(name)]<br> File "/usr/share/vdsm/storage/<wbr>multipath.py", line 158, in getDeviceSize<br> bs, phyBs = getDeviceBlockSizes(devName)<br> File "/usr/share/vdsm/storage/<wbr>multipath.py", line 150, in getDeviceBlockSizes<br> "queue", "logical_block_size")).read())<br>IOError: [Errno 2] No such file or directory: '/sys/block/sdb/queue/logical_<wbr>block_size'<br></div></blockquote><div><br></div><div><br></div><div>We now see a different error in master [1], which also indicates the hosts are in a problematic state: ( failing 'assign_hosts_network_label' test )</div><div><br></div><div><div>status: 409</div><div>reason: Conflict</div><div>detail: Cannot add Label. Operation can be performed only when Host status is Maintenance, Up, NonOperational.</div><div>-------------------- >> begin captured logging << --------------------</div></div><div><br></div><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3506/testReport/junit/(root)/006_network_by_label/assign_hosts_network_label/">http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/3506/testReport/junit/(root)/006_network_by_label/assign_hosts_network_label/</a></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 20, 2016 at 12:50 PM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Sun, Nov 20, 2016 at 1:42 PM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Sun, Nov 20, 2016 at 1:30 PM, Yaniv Kaul <span dir="ltr"><<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-m_6757589575781253264m_5281171221307826131m_1756134681961643914gmail-">On Sun, Nov 20, 2016 at 1:18 PM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">the test fails to run VM because no hosts are in UP state(?) [1], not sure it is related to the triggering patch[2]<div><br></div><div><div>status: 400</div><div>reason: Bad Request</div><div>detail: There are no hosts to use. Check that the cluster contains at least one host in Up state.</div></div><div><br></div><div>Thoughts? Shouldn't we fail the test earlier we hosts are not UP? </div></div></blockquote><div><br></div></span><div>Yes. It's more likely that we are picking the wrong host or so, but who knows - where are the engine and VDSM logs?</div></div></div></div></blockquote><div><br></div></span><div>A simple grep on the engine.log[1] finds serveral unrelated issues I'm not sure are reported, it's despairing to even begin...</div><div>That being said, I don't see the issue there. We may need better logging on the API level, to see what is being sent. Is it consistent?</div></div></div></div></blockquote><div><br></div></span><div>Just failed now the first time, I didn't see it before.</div><div><div class="gmail-m_6757589575781253264h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Y.</div><div><br></div><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3015/artifact/exported-artifacts/basic_suite_4.0.sh-el7/exported-artifacts/test_logs/basic-suite-4.0/post-004_basic_sanity.py/lago-basic-suite-4-0-engine/_var_log_ovirt-engine/engine.log" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_4.<wbr>0/3015/artifact/exported-artif<wbr>acts/basic_suite_4.0.sh-el7/ex<wbr>ported-artifacts/test_logs/bas<wbr>ic-suite-4.0/post-004_basic_<wbr>sanity.py/lago-basic-suite-4-0<wbr>-engine/_var_log_ovirt-engine/<wbr>engine.log</a> </div><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="gmail-m_6757589575781253264m_5281171221307826131m_1756134681961643914gmail-HOEnZb"><font color="#888888"><div>Y.</div></font></span><span class="gmail-m_6757589575781253264m_5281171221307826131m_1756134681961643914gmail-"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3015/testReport/junit/(root)/004_basic_sanity/vm_run/" target="_blank">http://jenkins.ovirt.org/j<wbr>ob/test-repo_ovirt_experimenta<wbr>l_4.0/3015/testReport/junit/(r<wbr>oot)/004_basic_sanity/vm_run/</a></div><div>[2] <a href="http://jenkins.ovirt.org/job/ovirt-engine_4.0_build-artifacts-el7-x86_64/1535/changes#detail" target="_blank">http://jenkins.ovirt.org/j<wbr>ob/ovirt-engine_4.0_build-arti<wbr>facts-el7-x86_64/1535/changes#<wbr>detail</a><br><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Nov 20, 2016 at 1:00 PM, <span dir="ltr"><<a href="mailto:jenkins@jenkins.phx.ovirt.org" target="_blank">jenkins@jenkins.phx.ovirt.org</a><wbr>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Build: <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_4.0/3015/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/t<wbr>est-repo_ovirt_experimental_4.<wbr>0/3015/</a>,<br>
Build Number: 3015,<br>
Build Status: FAILURE<br>______________________________<wbr>_________________<br>
Infra mailing list<br>
<a href="mailto:Infra@ovirt.org" target="_blank">Infra@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/infra</a><br>
<br></blockquote></div><span class="gmail-m_6757589575781253264m_5281171221307826131m_1756134681961643914gmail-m_7253519649333261386HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_6757589575781253264m_5281171221307826131m_1756134681961643914gmail-m_7253519649333261386m_36071486613131388gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</font></span></div>
</blockquote></span></div><br></div></div>
</blockquote></span></div><br></div></div>
</blockquote></div></div></div><div><div class="gmail-m_6757589575781253264h5"><br><br clear="all"><div><br></div>-- <br><div class="gmail-m_6757589575781253264m_5281171221307826131gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: <a href="tel:%2B972-9-7692018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div></div>