Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 01/08/2017 ] [add_secondary_storage_domains]

You should have all logs needed @/home/myoung/repos/github/ ovirt-system-tests/test_logs/basic-suite-4.1/post-002_ bootstrap.py/lago_logs

On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin(a)gmail.com&gt; wrote: > Full log is just as uninteresting: https://paste.f > edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA > I was actually hoping for the engine log. Y. > > On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > >> >> >> On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin(a)gmail.com&gt; >> wrote: >> >>> Thanks for clarifying, that makes sense in hindsight since its testing >>> things in their entirety. This is more of a rabbit hole for me as Im really >>> just trying to learn Lago by using the OST project (since I'll be >>> duplicating a lot of the setup). >>> >>> stdout logs from a run: https://pastebin.com/KBDaCCYp >>> >> >> Can you send engine log so we'll understand why the host installation >> failed? >> You should have all logs needed @/home/myoung/repos/github/ovi >> rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap >> .py/lago_logs >> >> Y. >> >> >>> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>> >>>> >>>> >>>> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin(a)gmail.com&gt; >>>> wrote: >>>> >>>>> Ill try to get some detailed log files later, but fwiw I'm not >>>>> running the hc suites (afaik, Im still getting inundated with the system >>>>> tests/lago). The link I used for Jenkins was just to try to pull the latest >>>>> 'passing' hash for ovirt-system-tests >>>>> >>>> >>>> I see, its a bit more complicated than that I'm afraid, the HASH you >>>> see is probably on for OST code itself, while the suites can fail on any >>>> oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more >>>> than just a single hash, but rather a list of RPMs and their versions, >>>> together with hash of OST ( for the tests themselves ) and also OS updates. >>>> >>>> >>>>> >>>>> The errors have all been from master branch on the basic suite for >>>>> 4.1 via: >>>>> >>>>> $ ./run_suite.sh basic-suite-4.1/ >>>>> >>>>> >>>> OK, then we need to investigate it and understand why add host fails >>>> on 4.1, please provide logs when possible. >>>> >>>> BTW, you can also try running the manual job [1], but you'll need to >>>> provide it with custom yum repo URL with your built artifacts to test your >>>> code. >>>> There is a section on it on the OST readthedocs page. >>>> >>>> [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job >>>> /ovirt-system-tests_manual/ >>>> >>>> >>>>> >>>>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>>> >>>>>> I see you run the 'hc' suite, which means hyperconverged. This suite >>>>>> is running an hosted engine on gluster storage, It's a more complex suite >>>>>> than the basic one, and prone to more errors, however it should still work, >>>>>> but if you don't require it specifically, I would recommend running the >>>>>> basic suite, which should be easier to debug and also quicker to run. >>>>>> >>>>>> I'm also adding hc maintainer so she can check, can you share the >>>>>> link to the Jenkins job you're running? Or the log files if you're running >>>>>> locally. >>>>>> >>>>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin(a)gmail.com&gt; wrote: >>>>>> >>>>>>> Actually I spoke too soon, still fails: >>>>>>> >>>>>>> + lago ovirt runtest /home/myoung/repos/github/ovir >>>>>>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >>>>>>> @ Run test: 002_bootstrap.py: >>>>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>>>>>> '^setup\\.py$'] >>>>>>> # print_api_ver: >>>>>>> # print_api_ver: Success (in 0:00:00) >>>>>>> # add_dc: >>>>>>> # add_dc: Success (in 0:00:43) >>>>>>> # add_cluster: >>>>>>> # add_cluster: Success (in 0:00:03) >>>>>>> # add_hosts: >>>>>>> dd >>>>>>> * Collect artifacts: >>>>>>> * Collect artifacts: Success (in 0:01:14) >>>>>>> # add_hosts: Success (in 0:16:36) >>>>>>> # Results located at /home/myoung/repos/github/ovir >>>>>>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j >>>>>>> unit.xml >>>>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>>>>>> Error occured, aborting >>>>>>> Traceback (most recent call last): >>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>>>>>> 360, in do_run >>>>>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >>>>>>> line 184, in do_run >>>>>>> self._do_run(**vars(args)) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, >>>>>>> in wrapper >>>>>>> return func(*args, **kwargs) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, >>>>>>> in wrapper >>>>>>> return func(*args, prefix=prefix, **kwargs) >>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>>>>>> 99, in do_ovirt_runtest >>>>>>> raise RuntimeError('Some tests failed') >>>>>>> RuntimeError: Some tests failed >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin(a)gmail.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Edit: reply-all >>>>>>>> >>>>>>>> It does, every time. >>>>>>>> I got it to pass by using the last passing revision from Jenkins >>>>>>>> (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from >>>>>>>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas >>>>>>>> ter/15/consoleFull >>>>>>>> >>>>>>>> Not sure if it's a true race condition, that build in Jenkins has >>>>>>>> failed consistently since #15 >>>>>>>> >>>>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young < >>>>>>>>> 3vilpenguin(a)gmail.com&gt; wrote: >>>>>>>>> >>>>>>>>>> What's the fix for this for those of us using ovirt-system-tests? >>>>>>>>>> >>>>>>>>>> Im trying to adapt some of the code for testing a third party >>>>>>>>>> tool, but master is still failing on 002_bootstrap >>>>>>>>>> >>>>>>>>> >>>>>>>>> It fails consistently? >>>>>>>>> AFAIK this is a race condition that happens maybe once a week, >>>>>>>>> can you share your logs? is it also failing on add_secondary_storage? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>>>>>>>> bzlotnik(a)redhat.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> I'm not sure it's related since the LSM test runs much later, >>>>>>>>>>> in 004 >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri(a)redhat.com&gt; >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Adding Allon & Benny. >>>>>>>>>>>> >>>>>>>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>>>>>>> we've been handling in [1]? >>>>>>>>>>>> If this is the case, we agreed to disable the test next time >>>>>>>>>>>> it fails, as the current workarounds with sleep isn't enough. >>>>>>>>>>>> >>>>>>>>>>>> Can you confirm this is the case, and so we'll have to disable >>>>>>>>>>>> this test until one of the RFEs described in [1] is merged? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>>>>>>>> bkorren(a)redhat.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>>>>>>>> bkorren(a)redhat.com&gt; wrote: >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>>>>>>>> ] >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Link to suspected patches: >>>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > This patch adds missing log when resizing an online disk - >>>>>>>>>>>>> why do you think >>>>>>>>>>>>> > it is related to the failure? >>>>>>>>>>>>> >>>>>>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>>>>>> (The test was equivalent to running the manual job with just >>>>>>>>>>>>> this patch) >>>>>>>>>>>>> >>>>>>>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>>>>>>> conditions. >>>>>>>>>>>>> Since the code in the patch seemed to be related to storage I >>>>>>>>>>>>> didn't >>>>>>>>>>>>> want to just assume that. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Barak Korren >>>>>>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>>>>>> Red Hat EMEA >>>>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Devel mailing list >>>>>>>>>>>>> Devel(a)ovirt.org >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Eyal edri >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ASSOCIATE MANAGER >>>>>>>>>>>> >>>>>>>>>>>> RHV DevOps >>>>>>>>>>>> >>>>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>>>>> <https://redhat.com/trusted> >>>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list >>>>>>>>>>> Devel(a)ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Eyal edri >>>>>>>>> >>>>>>>>> >>>>>>>>> ASSOCIATE MANAGER >>>>>>>>> >>>>>>>>> RHV DevOps >>>>>>>>> >>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>> >>>>>>>>> >>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>> <https://redhat.com/trusted> >>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Eyal edri >>>> >>>> >>>> ASSOCIATE MANAGER >>>> >>>> RHV DevOps >>>> >>>> EMEA VIRTUALIZATION R&D >>>> >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>> <https://redhat.com/trusted> >>>> phone: +972-9-7692018 <+972%209-769-2018> >>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> >