Re: [ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 01/08/2017 ] [add_secondary_storage_domains]

You should have all logs needed @/home/myoung/repos/github/ovi > rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. Heres all logs for basic suite 4.1 that I know how to find: myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲

$ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; done Uploading (1355.0KiB)... https://da.gd/OIlR -> https://paste.fedoraproject.org/paste/ 9Xn2RuDgF2pY2I0CouBbGg/ Uploading (1154.0KiB)... https://da.gd/0Cti -> https://paste.fedoraproject.org/paste/ uJqvAiDSblnmN8OA6i5tXA/ Uploading (1377.8KiB)... https://da.gd/GpDIf -> https://paste.fedoraproject.org/paste/~ Cn6IJgbvateJfFulDNFCg/ Uploading (1176.7KiB)... https://da.gd/skhBy -> https://paste.fedoraproject.org/paste/ HIpEVm4TRPpH5nJCOyUfWA/ Uploading (1397.1KiB)... https://da.gd/fEtF -> https://paste.fedoraproject.org/paste/cuycP~u9D~ gJKNPsQ5NH2g/ Uploading (1198.1KiB)... https://da.gd/1zNm -> https://paste.fedoraproject. org/paste/5~yEDwCSr~tdxHT~XUaLPg/ On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > > > On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin(a)gmail.com&gt; > wrote: > >> Full log is just as uninteresting: https://paste.f >> edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA >> > > I was actually hoping for the engine log. > Y. > > >> >> On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >> >>> >>> >>> On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin(a)gmail.com&gt; >>> wrote: >>> >>>> Thanks for clarifying, that makes sense in hindsight since its testing >>>> things in their entirety. This is more of a rabbit hole for me as Im really >>>> just trying to learn Lago by using the OST project (since I'll be >>>> duplicating a lot of the setup). >>>> >>>> stdout logs from a run: https://pastebin.com/KBDaCCYp >>>> >>> >>> Can you send engine log so we'll understand why the host installation >>> failed? >>> You should have all logs needed @/home/myoung/repos/github/ovi >>> rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap >>> .py/lago_logs >>> >>> Y. >>> >>> >>>> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>> >>>>> >>>>> >>>>> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Ill try to get some detailed log files later, but fwiw I'm not >>>>>> running the hc suites (afaik, Im still getting inundated with the system >>>>>> tests/lago). The link I used for Jenkins was just to try to pull the latest >>>>>> 'passing' hash for ovirt-system-tests >>>>>> >>>>> >>>>> I see, its a bit more complicated than that I'm afraid, the HASH you >>>>> see is probably on for OST code itself, while the suites can fail on any >>>>> oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more >>>>> than just a single hash, but rather a list of RPMs and their versions, >>>>> together with hash of OST ( for the tests themselves ) and also OS updates. >>>>> >>>>> >>>>>> >>>>>> The errors have all been from master branch on the basic suite for >>>>>> 4.1 via: >>>>>> >>>>>> $ ./run_suite.sh basic-suite-4.1/ >>>>>> >>>>>> >>>>> OK, then we need to investigate it and understand why add host fails >>>>> on 4.1, please provide logs when possible. >>>>> >>>>> BTW, you can also try running the manual job [1], but you'll need to >>>>> provide it with custom yum repo URL with your built artifacts to test your >>>>> code. >>>>> There is a section on it on the OST readthedocs page. >>>>> >>>>> [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job >>>>> /ovirt-system-tests_manual/ >>>>> >>>>> >>>>>> >>>>>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> I see you run the 'hc' suite, which means hyperconverged. This >>>>>>> suite is running an hosted engine on gluster storage, It's a more complex >>>>>>> suite than the basic one, and prone to more errors, however it should still >>>>>>> work, but if you don't require it specifically, I would recommend running >>>>>>> the basic suite, which should be easier to debug and also quicker to run. >>>>>>> >>>>>>> I'm also adding hc maintainer so she can check, can you share the >>>>>>> link to the Jenkins job you're running? Or the log files if you're running >>>>>>> locally. >>>>>>> >>>>>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin(a)gmail.com&gt; wrote: >>>>>>> >>>>>>>> Actually I spoke too soon, still fails: >>>>>>>> >>>>>>>> + lago ovirt runtest /home/myoung/repos/github/ovir >>>>>>>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >>>>>>>> @ Run test: 002_bootstrap.py: >>>>>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>>>>>>> '^setup\\.py$'] >>>>>>>> # print_api_ver: >>>>>>>> # print_api_ver: Success (in 0:00:00) >>>>>>>> # add_dc: >>>>>>>> # add_dc: Success (in 0:00:43) >>>>>>>> # add_cluster: >>>>>>>> # add_cluster: Success (in 0:00:03) >>>>>>>> # add_hosts: >>>>>>>> dd >>>>>>>> * Collect artifacts: >>>>>>>> * Collect artifacts: Success (in 0:01:14) >>>>>>>> # add_hosts: Success (in 0:16:36) >>>>>>>> # Results located at /home/myoung/repos/github/ovir >>>>>>>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j >>>>>>>> unit.xml >>>>>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>>>>>>> Error occured, aborting >>>>>>>> Traceback (most recent call last): >>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>>>>>>> 360, in do_run >>>>>>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>>>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >>>>>>>> line 184, in do_run >>>>>>>> self._do_run(**vars(args)) >>>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>>>> 501, in wrapper >>>>>>>> return func(*args, **kwargs) >>>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>>>> 512, in wrapper >>>>>>>> return func(*args, prefix=prefix, **kwargs) >>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>>>>>>> 99, in do_ovirt_runtest >>>>>>>> raise RuntimeError('Some tests failed') >>>>>>>> RuntimeError: Some tests failed >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin(a)gmail.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Edit: reply-all >>>>>>>>> >>>>>>>>> It does, every time. >>>>>>>>> I got it to pass by using the last passing revision from Jenkins >>>>>>>>> (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from >>>>>>>>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas >>>>>>>>> ter/15/consoleFull >>>>>>>>> >>>>>>>>> Not sure if it's a true race condition, that build in Jenkins has >>>>>>>>> failed consistently since #15 >>>>>>>>> >>>>>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri(a)redhat.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young < >>>>>>>>>> 3vilpenguin(a)gmail.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> What's the fix for this for those of us using >>>>>>>>>>> ovirt-system-tests? >>>>>>>>>>> >>>>>>>>>>> Im trying to adapt some of the code for testing a third party >>>>>>>>>>> tool, but master is still failing on 002_bootstrap >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It fails consistently? >>>>>>>>>> AFAIK this is a race condition that happens maybe once a week, >>>>>>>>>> can you share your logs? is it also failing on add_secondary_storage? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>>>>>>>>> bzlotnik(a)redhat.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> I'm not sure it's related since the LSM test runs much later, >>>>>>>>>>>> in 004 >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri(a)redhat.com&gt; >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Adding Allon & Benny. >>>>>>>>>>>>> >>>>>>>>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>>>>>>>> we've been handling in [1]? >>>>>>>>>>>>> If this is the case, we agreed to disable the test next time >>>>>>>>>>>>> it fails, as the current workarounds with sleep isn't enough. >>>>>>>>>>>>> >>>>>>>>>>>>> Can you confirm this is the case, and so we'll have to >>>>>>>>>>>>> disable this test until one of the RFEs described in [1] is merged? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>>>>>>>>> bkorren(a)redhat.com&gt; wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer(a)redhat.com&gt; >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>>>>>>>>> bkorren(a)redhat.com&gt; wrote: >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>>>>>>>>> ] >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> Link to suspected patches: >>>>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > This patch adds missing log when resizing an online disk - >>>>>>>>>>>>>> why do you think >>>>>>>>>>>>>> > it is related to the failure? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>>>>>>> (The test was equivalent to running the manual job with just >>>>>>>>>>>>>> this patch) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>>>>>>>> conditions. >>>>>>>>>>>>>> Since the code in the patch seemed to be related to storage >>>>>>>>>>>>>> I didn't >>>>>>>>>>>>>> want to just assume that. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Barak Korren >>>>>>>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>>>>>>> Red Hat EMEA >>>>>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Devel mailing list >>>>>>>>>>>>>> Devel(a)ovirt.org >>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Eyal edri >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ASSOCIATE MANAGER >>>>>>>>>>>>> >>>>>>>>>>>>> RHV DevOps >>>>>>>>>>>>> >>>>>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>>>>>> <https://redhat.com/trusted> >>>>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Devel mailing list >>>>>>>>>>>> Devel(a)ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Eyal edri >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ASSOCIATE MANAGER >>>>>>>>>> >>>>>>>>>> RHV DevOps >>>>>>>>>> >>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>>> <https://redhat.com/trusted> >>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Eyal edri >>>>> >>>>> >>>>> ASSOCIATE MANAGER >>>>> >>>>> RHV DevOps >>>>> >>>>> EMEA VIRTUALIZATION R&D >>>>> >>>>> >>>>> Red Hat EMEA <https://www.redhat.com/> >>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>> <https://redhat.com/trusted> >>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >> >