<div dir="ltr">Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).<div><br></div><div>stdout logs from a run: <a href="https://pastebin.com/KBDaCCYp">https://pastebin.com/KBDaCCYp</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <span dir="ltr"><<a href="mailto:3vilpenguin@gmail.com" target="_blank">3vilpenguin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests</div></blockquote><div><br></div></span><div>I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/<wbr>etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates. </div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>The errors have all been from master branch on the basic suite for 4.1 via:</div><div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>$ ./run_suite.sh basic-suite-4.1/<br></div></blockquote></div></div></blockquote><div><br></div></span><div>OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.</div><div><br></div><div>BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code.</div><div>There is a section on it on the OST readthedocs page.</div><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/" target="_blank">http://jenkins.ovirt.org/<wbr>view/oVirt%20system%20tests/<wbr>job/ovirt-system-tests_manual/</a></div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div></div></blockquote></div></div><div class="m_4865404260528103245gmail-HOEnZb"><div class="m_4865404260528103245gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.<div dir="auto"><br></div><div dir="auto">I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.</div></div><div class="m_4865404260528103245gmail-m_-3951477646076186353HOEnZb"><div class="m_4865404260528103245gmail-m_-3951477646076186353h5"><div class="gmail_extra"><br><div class="gmail_quote">On Aug 14, 2017 06:00, "Marc Young" <<a href="mailto:3vilpenguin@gmail.com" target="_blank">3vilpenguin@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Actually I spoke too soon, still fails:<div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>+ lago ovirt runtest /home/myoung/repos/github/ovir<wbr>t-system-tests/vagrant/test-sc<wbr>enarios/002_bootstrap.py</div><div>@ Run test: 002_bootstrap.py:</div><div>nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']</div><div> # print_api_ver:</div><div> # print_api_ver: Success (in 0:00:00)</div><div> # add_dc:</div><div> # add_dc: Success (in 0:00:43)</div><div> # add_cluster:</div><div> # add_cluster: Success (in 0:00:03)</div><div> # add_hosts:</div><div>dd</div><div> * Collect artifacts:</div><div> * Collect artifacts: Success (in 0:01:14)</div><div> # add_hosts: Success (in 0:16:36)</div><div> # Results located at /home/myoung/repos/github/ovir<wbr>t-system-tests/deployment-vagr<wbr>ant/default/002_bootstrap.py.j<wbr>unit.xml</div><div>@ Run test: 002_bootstrap.py: Success (in 0:17:26)</div><div>Error occured, aborting</div><div>Traceback (most recent call last):</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/cmd.py", line 360, in do_run</div><div> self.cli_plugins[args.ovirtver<wbr>b].do_run(args)</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/lago/plugins/cli.py", line 184, in do_run</div><div> self._do_run(**vars(args))</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 501, in wrapper</div><div> return func(*args, **kwargs)</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/lago/utils.py", line 512, in wrapper</div><div> return func(*args, prefix=prefix, **kwargs)</div><div> File "/usr/lib/python2.7/site-packa<wbr>ges/ovirtlago/cmd.py", line 99, in do_ovirt_runtest</div><div> raise RuntimeError('Some tests failed')</div><div>RuntimeError: Some tests failed</div></div><div><br></div><div><br></div></blockquote><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <span dir="ltr"><<a href="mailto:3vilpenguin@gmail.com" target="_blank">3vilpenguin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Edit: reply-all<span><div><br></div><div><span style="font-size:12.8px">It does, every time.</span><div style="font-size:12.8px">I got it to pass by using the last passing revision from Jenkins (hash <span style="color:rgb(51,51,51);font-size:15px;white-space:pre-wrap">98ae6d0b452d098f2703a197<wbr>deb082a091bba837 ), noted from </span><font color="#333333"><span style="font-size:15px;white-space:pre-wrap"><a href="http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-master/15/consoleFull" target="_blank">http://jenkins.ovirt.org/job/s<wbr>ystem-tests_hc-basic-suite-mas<wbr>ter/15/consoleFull</a> </span></font></div></div><div style="font-size:12.8px"><font color="#333333"><span style="font-size:15px;white-space:pre-wrap"><br></span></font></div></span><div><span style="font-size:12.8px">Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15</span></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br></span><div><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <span dir="ltr"><<a href="mailto:3vilpenguin@gmail.com" target="_blank">3vilpenguin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">What's the fix for this for those of us using ovirt-system-tests?<div><br><div>Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap</div></div></div></blockquote><div><br></div></span><div>It fails consistently? </div><div>AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage? </div><div><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050h5"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012HOEnZb"><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <span dir="ltr"><<a href="mailto:bzlotnik@redhat.com" target="_blank">bzlotnik@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm not sure it's related since the LSM test runs much later, in 004</div><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012m_-3910692678509222295HOEnZb"><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012m_-3910692678509222295h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <span dir="ltr"><<a href="mailto:eedri@redhat.com" target="_blank">eedri@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Adding Allon & Benny.<div><br></div><div>Is it possible to verify this is related to the LSM issue we've been handling in [1]? </div><div>If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.</div><div><br></div><div>Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?</div><div><br></div><div><br></div><div>[1] <a href="https://gerrit.ovirt.org/#/c/78613/" target="_blank">https://gerrit.ovirt.org/#<wbr>/c/78613/</a></div></div><div class="gmail_extra"><div><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012m_-3910692678509222295m_-4594783272345111135h5"><br><div class="gmail_quote">On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <span dir="ltr"><<a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On 1 August 2017 at 14:39, Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>> wrote:<br>
><br>
><br>
> On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <<a href="mailto:bkorren@redhat.com" target="_blank">bkorren@redhat.com</a>> wrote:<br>
>><br>
>> Test failed: [ 002_bootstrap.add_secondary_st<wbr>orage_domains ]<br>
>><br>
>> Link to suspected patches:<br>
>> <a href="https://gerrit.ovirt.org/#/c/79974" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/7<wbr>9974</a><br>
><br>
><br>
> This patch adds missing log when resizing an online disk - why do you think<br>
> it is related to the failure?<br>
<br>
</span>Because it is the only patch participating in the test.<br>
(The test was equivalent to running the manual job with just this patch)<br>
<br>
Then again this may also one of the usual SD testing race conditions.<br>
Since the code in the patch seemed to be related to storage I didn't<br>
want to just assume that.<br>
<div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012m_-3910692678509222295m_-4594783272345111135m_4959142784202561159HOEnZb"><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012m_-3910692678509222295m_-4594783272345111135m_4959142784202561159h5"><br>
<br>
--<br>
Barak Korren<br>
RHV DevOps team , RHCE, RHCi<br>
Red Hat EMEA<br>
<a href="http://redhat.com" rel="noreferrer" target="_blank">redhat.com</a> | TRIED. TESTED. TRUSTED. | <a href="http://redhat.com/trusted" rel="noreferrer" target="_blank">redhat.com/trusted</a><br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div>-- <br><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012m_-3910692678509222295m_-4594783272345111135m_4959142784202561159gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">ASSOCIATE MANAGER</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat EMEA</a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></td></tr></tbody></table></div><div>phone: <a href="tel:+972%209-769-2018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div></div></div>
</div>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br></blockquote></div><br></div>
</div></div></blockquote></div></div></div><div><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050h5"><br><br clear="all"><div><br></div>-- <br><div class="m_4865404260528103245gmail-m_-3951477646076186353m_8511336199060387257m_-2911890537313925973m_4451550972513183050m_5979519333773425012gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">ASSOCIATE MANAGER</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat EMEA</a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></td></tr></tbody></table></div><div>phone: <a href="tel:+972%209-769-2018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div></div></div>
</div></div></div></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</blockquote></div></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div class="m_4865404260528103245gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><p style="font-family:overpass,sans-serif;margin:0px;padding:0px;font-size:14px;text-transform:uppercase;font-weight:bold"><font color="#cc0000">Eyal edri</font></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><br></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">ASSOCIATE MANAGER</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">RHV DevOps</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase">EMEA VIRTUALIZATION R&D</p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><br></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat EMEA</a></p><table border="0" style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a></td><td style="font-size:10px"><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></td></tr></tbody></table></div><div>phone: <a href="tel:+972%209-769-2018" value="+97297692018" target="_blank">+972-9-7692018</a><br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div></div></div>
</div></div></div></div>
</blockquote></div><br></div>