[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 01/08/2017 ] [add_secondary_storage_domains]

Eyal Edri eedri at redhat.com
Mon Aug 14 18:11:07 UTC 2017 

On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin at gmail.com> wrote:

> You should have all logs needed @/home/myoung/repos/github/ovi
>> rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs
>
>
> The one I pasted into fpaste was the only one in that directory, if there
> are more logs Im just unsure where they live.
> Heres all logs for basic suite 4.1 that I know how to find:
>
> myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
>
>
The engine and hosts logs are usually under the test_logs, which should be
in the same subdir as lago_logs directory.
Also, can you verify you have the latest python v4 SDK installled?

rpm -q python-ovirt-engine-sdk4

If you don't have it installed, you can get it directly from the build job
( which acts also as yum repo ):

el7 build -
http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_4.1_build-artifacts-el7-x86_64/lastSuccessfulBuild/artifact/exported-artifacts/

fc26 build is available only for master [1] but should also work.

There are also fc24/25 builds, just follow the same pattern to find them on
Jenkins.


[1]
http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_master_build-artifacts-fc26-x86_64/lastSuccessfulBuild/artifact/exported-artifacts/



> $ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat
> $i/*.log | fpaste; done
> Uploading (1355.0KiB)...
> https://da.gd/OIlR -> https://paste.fedoraproject.org/paste/
> 9Xn2RuDgF2pY2I0CouBbGg/
> Uploading (1154.0KiB)...
> https://da.gd/0Cti -> https://paste.fedoraproject.org/paste/
> uJqvAiDSblnmN8OA6i5tXA/
> Uploading (1377.8KiB)...
> https://da.gd/GpDIf -> https://paste.fedoraproject.org/paste/~
> Cn6IJgbvateJfFulDNFCg/
> Uploading (1176.7KiB)...
> https://da.gd/skhBy -> https://paste.fedoraproject.org/paste/
> HIpEVm4TRPpH5nJCOyUfWA/
> Uploading (1397.1KiB)...
> https://da.gd/fEtF -> https://paste.fedoraproject.org/paste/cuycP~u9D~
> gJKNPsQ5NH2g/
> Uploading (1198.1KiB)...
> https://da.gd/1zNm -> https://paste.fedoraproject.
> org/paste/5~yEDwCSr~tdxHT~XUaLPg/
>
>
> On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>
>>
>>
>> On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin at gmail.com>
>> wrote:
>>
>>> Full log is just as uninteresting: https://paste.f
>>> edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
>>>
>>
>> I was actually hoping for the engine log.
>> Y.
>>
>>
>>>
>>> On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin at gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks for clarifying, that makes sense in hindsight since its testing
>>>>> things in their entirety. This is more of a rabbit hole for me as Im really
>>>>> just trying to learn Lago by using the OST project (since I'll be
>>>>> duplicating a lot of the setup).
>>>>>
>>>>> stdout logs from a run: https://pastebin.com/KBDaCCYp
>>>>>
>>>>
>>>> Can you send engine log so we'll understand why the host installation
>>>> failed?
>>>> You should have all logs needed @/home/myoung/repos/github/ovi
>>>> rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap
>>>> .py/lago_logs
>>>>
>>>> Y.
>>>>
>>>>
>>>>> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri at redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Ill try to get some detailed log files later, but fwiw I'm not
>>>>>>> running the hc suites (afaik, Im still getting inundated with the system
>>>>>>> tests/lago). The link I used for Jenkins was just to try to pull the latest
>>>>>>> 'passing' hash for ovirt-system-tests
>>>>>>>
>>>>>>
>>>>>> I see, its a bit more complicated than that I'm afraid, the HASH you
>>>>>> see is probably on for OST code itself, while the suites can fail on any
>>>>>> oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more
>>>>>> than just a single hash, but rather a list of RPMs and their versions,
>>>>>> together with hash of OST ( for the tests themselves ) and also OS updates.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> The errors have all been from master branch on the basic suite for
>>>>>>> 4.1 via:
>>>>>>>
>>>>>>> $ ./run_suite.sh basic-suite-4.1/
>>>>>>>
>>>>>>>
>>>>>> OK, then we need to investigate it and understand why add host fails
>>>>>> on 4.1, please provide logs when possible.
>>>>>>
>>>>>> BTW, you can also try running the manual job [1], but you'll need to
>>>>>> provide it with custom yum repo URL with your built artifacts to test your
>>>>>> code.
>>>>>> There is a section on it on the OST readthedocs page.
>>>>>>
>>>>>> [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job
>>>>>> /ovirt-system-tests_manual/
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I see you run the 'hc' suite, which means hyperconverged. This
>>>>>>>> suite is running an hosted engine on gluster storage, It's a more complex
>>>>>>>> suite than the basic one, and prone to more errors, however it should still
>>>>>>>> work, but if you don't require it specifically, I would recommend running
>>>>>>>> the basic suite, which should be easier to debug and also quicker to run.
>>>>>>>>
>>>>>>>> I'm also adding hc maintainer so she can check, can you share the
>>>>>>>> link to the Jenkins job you're running? Or the log files if you're running
>>>>>>>> locally.
>>>>>>>>
>>>>>>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Actually I spoke too soon, still fails:
>>>>>>>>>
>>>>>>>>> + lago ovirt runtest /home/myoung/repos/github/ovir
>>>>>>>>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py
>>>>>>>>> @ Run test: 002_bootstrap.py:
>>>>>>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_',
>>>>>>>>> '^setup\\.py$']
>>>>>>>>>   # print_api_ver:
>>>>>>>>>   # print_api_ver: Success (in 0:00:00)
>>>>>>>>>   # add_dc:
>>>>>>>>>   # add_dc: Success (in 0:00:43)
>>>>>>>>>   # add_cluster:
>>>>>>>>>   # add_cluster: Success (in 0:00:03)
>>>>>>>>>   # add_hosts:
>>>>>>>>> dd
>>>>>>>>>     * Collect artifacts:
>>>>>>>>>     * Collect artifacts: Success (in 0:01:14)
>>>>>>>>>   # add_hosts: Success (in 0:16:36)
>>>>>>>>>   # Results located at /home/myoung/repos/github/ovir
>>>>>>>>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j
>>>>>>>>> unit.xml
>>>>>>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26)
>>>>>>>>> Error occured, aborting
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line
>>>>>>>>> 360, in do_run
>>>>>>>>>     self.cli_plugins[args.ovirtverb].do_run(args)
>>>>>>>>>   File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py",
>>>>>>>>> line 184, in do_run
>>>>>>>>>     self._do_run(**vars(args))
>>>>>>>>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line
>>>>>>>>> 501, in wrapper
>>>>>>>>>     return func(*args, **kwargs)
>>>>>>>>>   File "/usr/lib/python2.7/site-packages/lago/utils.py", line
>>>>>>>>> 512, in wrapper
>>>>>>>>>     return func(*args, prefix=prefix, **kwargs)
>>>>>>>>>   File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line
>>>>>>>>> 99, in do_ovirt_runtest
>>>>>>>>>     raise RuntimeError('Some tests failed')
>>>>>>>>> RuntimeError: Some tests failed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin at gmail.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Edit: reply-all
>>>>>>>>>>
>>>>>>>>>> It does, every time.
>>>>>>>>>> I got it to pass by using the last passing revision from Jenkins
>>>>>>>>>> (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from
>>>>>>>>>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas
>>>>>>>>>> ter/15/consoleFull
>>>>>>>>>>
>>>>>>>>>> Not sure if it's a true race condition, that build in Jenkins has
>>>>>>>>>> failed consistently since #15
>>>>>>>>>>
>>>>>>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri at redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <
>>>>>>>>>>> 3vilpenguin at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> What's the fix for this for those of us using
>>>>>>>>>>>> ovirt-system-tests?
>>>>>>>>>>>>
>>>>>>>>>>>> Im trying to adapt some of the code for testing a third party
>>>>>>>>>>>> tool, but master is still failing on 002_bootstrap
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It fails consistently?
>>>>>>>>>>> AFAIK this is a race condition that happens maybe once a week,
>>>>>>>>>>> can you share your logs? is it also failing on add_secondary_storage?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <
>>>>>>>>>>>> bzlotnik at redhat.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure it's related since the LSM test runs much later,
>>>>>>>>>>>>> in 004
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri at redhat.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Adding Allon & Benny.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is it possible to verify this is related to the LSM issue
>>>>>>>>>>>>>> we've been handling in [1]?
>>>>>>>>>>>>>> If this is the case, we agreed to disable the test next time
>>>>>>>>>>>>>> it fails, as the current workarounds with sleep isn't enough.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you confirm this is the case, and so we'll have to
>>>>>>>>>>>>>> disable this test until one of the RFEs described in [1] is merged?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <
>>>>>>>>>>>>>> bkorren at redhat.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer at redhat.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <
>>>>>>>>>>>>>>> bkorren at redhat.com> wrote:
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains
>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>> >> Link to suspected patches:
>>>>>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > This patch adds missing log when resizing an online disk -
>>>>>>>>>>>>>>> why do you think
>>>>>>>>>>>>>>> > it is related to the failure?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Because it is the only patch participating in the test.
>>>>>>>>>>>>>>> (The test was equivalent to running the manual job with just
>>>>>>>>>>>>>>> this patch)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Then again this may also one of the usual SD testing race
>>>>>>>>>>>>>>> conditions.
>>>>>>>>>>>>>>> Since the code in the patch seemed to be related to storage
>>>>>>>>>>>>>>> I didn't
>>>>>>>>>>>>>>> want to just assume that.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Barak Korren
>>>>>>>>>>>>>>> RHV DevOps team , RHCE, RHCi
>>>>>>>>>>>>>>> Red Hat EMEA
>>>>>>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Devel mailing list
>>>>>>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Eyal edri
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ASSOCIATE MANAGER
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> RHV DevOps
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>>>>>>>>>> <https://redhat.com/trusted>
>>>>>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Devel mailing list
>>>>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Eyal edri
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ASSOCIATE MANAGER
>>>>>>>>>>>
>>>>>>>>>>> RHV DevOps
>>>>>>>>>>>
>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>>>>>>> <https://redhat.com/trusted>
>>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Eyal edri
>>>>>>
>>>>>>
>>>>>> ASSOCIATE MANAGER
>>>>>>
>>>>>> RHV DevOps
>>>>>>
>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>
>>>>>>
>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>> <https://redhat.com/trusted>
>>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Devel mailing list
>>>>> Devel at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>
>>>>
>>>>
>>>
>>
>


-- 

Eyal edri


ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D


Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170814/c4e65cf4/attachment-0001.html>

More information about the Devel mailing list