Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 01/08/2017 ] [add_secondary_storage_domains]

On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin(a)gmail.com&gt; wrote: > Ill try to get some detailed log files later, but fwiw I'm not running > the hc suites (afaik, Im still getting inundated with the system > tests/lago). The link I used for Jenkins was just to try to pull the latest > 'passing' hash for ovirt-system-tests > I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates. > > The errors have all been from master branch on the basic suite for 4.1 > via: > > $ ./run_suite.sh basic-suite-4.1/ > > OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible. BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code. There is a section on it on the OST readthedocs page. [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/ job/ovirt-system-tests_manual/ > > On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: > >> I see you run the 'hc' suite, which means hyperconverged. This suite is >> running an hosted engine on gluster storage, It's a more complex suite than >> the basic one, and prone to more errors, however it should still work, but >> if you don't require it specifically, I would recommend running the basic >> suite, which should be easier to debug and also quicker to run. >> >> I'm also adding hc maintainer so she can check, can you share the link >> to the Jenkins job you're running? Or the log files if you're running >> locally. >> >> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin(a)gmail.com&gt; wrote: >> >>> Actually I spoke too soon, still fails: >>> >>> + lago ovirt runtest /home/myoung/repos/github/ovir >>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >>> @ Run test: 002_bootstrap.py: >>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>> '^setup\\.py$'] >>> # print_api_ver: >>> # print_api_ver: Success (in 0:00:00) >>> # add_dc: >>> # add_dc: Success (in 0:00:43) >>> # add_cluster: >>> # add_cluster: Success (in 0:00:03) >>> # add_hosts: >>> dd >>> * Collect artifacts: >>> * Collect artifacts: Success (in 0:01:14) >>> # add_hosts: Success (in 0:16:36) >>> # Results located at /home/myoung/repos/github/ovir >>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml >>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>> Error occured, aborting >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, >>> in do_run >>> self.cli_plugins[args.ovirtverb].do_run(args) >>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line >>> 184, in do_run >>> self._do_run(**vars(args)) >>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in >>> wrapper >>> return func(*args, **kwargs) >>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in >>> wrapper >>> return func(*args, prefix=prefix, **kwargs) >>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, >>> in do_ovirt_runtest >>> raise RuntimeError('Some tests failed') >>> RuntimeError: Some tests failed >>> >>> >>> >>> >>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin(a)gmail.com&gt; >>> wrote: >>> >>>> Edit: reply-all >>>> >>>> It does, every time. >>>> I got it to pass by using the last passing revision from Jenkins (hash >>>> 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from >>>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas >>>> ter/15/consoleFull >>>> >>>> Not sure if it's a true race condition, that build in Jenkins has >>>> failed consistently since #15 >>>> >>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>> >>>>> >>>>> >>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> What's the fix for this for those of us using ovirt-system-tests? >>>>>> >>>>>> Im trying to adapt some of the code for testing a third party tool, >>>>>> but master is still failing on 002_bootstrap >>>>>> >>>>> >>>>> It fails consistently? >>>>> AFAIK this is a race condition that happens maybe once a week, can >>>>> you share your logs? is it also failing on add_secondary_storage? >>>>> >>>>> >>>>> >>>>>> >>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> I'm not sure it's related since the LSM test runs much later, in 004 >>>>>>> >>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> Adding Allon & Benny. >>>>>>>> >>>>>>>> Is it possible to verify this is related to the LSM issue we've >>>>>>>> been handling in [1]? >>>>>>>> If this is the case, we agreed to disable the test next time it >>>>>>>> fails, as the current workarounds with sleep isn't enough. >>>>>>>> >>>>>>>> Can you confirm this is the case, and so we'll have to disable >>>>>>>> this test until one of the RFEs described in [1] is merged? >>>>>>>> >>>>>>>> >>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>> >>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren(a)redhat.com&gt; >>>>>>>>> wrote: >>>>>>>>> >> >>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] >>>>>>>>> >> >>>>>>>>> >> Link to suspected patches: >>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > This patch adds missing log when resizing an online disk - why >>>>>>>>> do you think >>>>>>>>> > it is related to the failure? >>>>>>>>> >>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>> (The test was equivalent to running the manual job with just this >>>>>>>>> patch) >>>>>>>>> >>>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>>> conditions. >>>>>>>>> Since the code in the patch seemed to be related to storage I >>>>>>>>> didn't >>>>>>>>> want to just assume that. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Barak Korren >>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>> Red Hat EMEA >>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list >>>>>>>>> Devel(a)ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Eyal edri >>>>>>>> >>>>>>>> >>>>>>>> ASSOCIATE MANAGER >>>>>>>> >>>>>>>> RHV DevOps >>>>>>>> >>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>> >>>>>>>> >>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>> <https://redhat.com/trusted> >>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel(a)ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Eyal edri >>>>> >>>>> >>>>> ASSOCIATE MANAGER >>>>> >>>>> RHV DevOps >>>>> >>>>> EMEA VIRTUALIZATION R&D >>>>> >>>>> >>>>> Red Hat EMEA <https://www.redhat.com/> >>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>> <https://redhat.com/trusted> >>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>> >>>> >>>> >>> > -- Eyal edri ASSOCIATE MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)