[ OST Failure Report ] [ oVirt master ] [ 01/08/2017 ] [add_secondary_storage_domains]
Test failed: [ 002_bootstrap.add_secondary_storage_domains ] Link to suspected patches: https://gerrit.ovirt.org/#/c/79974 Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1597/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1597/artifact/... Error snippet from log: <error> Error: Fault reason is "Operation Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]". HTTP response code is 400. </error> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> wrote:
Test failed: [ 002_bootstrap.add_secondary_storage_domains ]
Link to suspected patches: https://gerrit.ovirt.org/#/c/79974
This patch adds missing log when resizing an online disk - why do you think it is related to the failure?
Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1597/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1597/artifact/...
Error snippet from log:
<error>
Error: Fault reason is "Operation Failed". Fault detail is "[Storage domain cannot be reached. Please ensure it is accessible from the host(s).]". HTTP response code is 400.
</error>
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> wrote:
Test failed: [ 002_bootstrap.add_secondary_storage_domains ]
Link to suspected patches: https://gerrit.ovirt.org/#/c/79974
This patch adds missing log when resizing an online disk - why do you think it is related to the failure?
Because it is the only patch participating in the test. (The test was equivalent to running the manual job with just this patch) Then again this may also one of the usual SD testing race conditions. Since the code in the patch seemed to be related to storage I didn't want to just assume that. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
Adding Allon & Benny. Is it possible to verify this is related to the LSM issue we've been handling in [1]? If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough. Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged? [1] https://gerrit.ovirt.org/#/c/78613/ On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> wrote:
Test failed: [ 002_bootstrap.add_secondary_storage_domains ]
Link to suspected patches: https://gerrit.ovirt.org/#/c/79974
This patch adds missing log when resizing an online disk - why do you
think
it is related to the failure?
Because it is the only patch participating in the test. (The test was equivalent to running the manual job with just this patch)
Then again this may also one of the usual SD testing race conditions. Since the code in the patch seemed to be related to storage I didn't want to just assume that.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal edri ASSOCIATE MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
I'm not sure it's related since the LSM test runs much later, in 004 On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
Adding Allon & Benny.
Is it possible to verify this is related to the LSM issue we've been handling in [1]? If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.
Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?
[1] https://gerrit.ovirt.org/#/c/78613/
On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> wrote:
Test failed: [ 002_bootstrap.add_secondary_storage_domains ]
Link to suspected patches: https://gerrit.ovirt.org/#/c/79974
This patch adds missing log when resizing an online disk - why do you
think
it is related to the failure?
Because it is the only patch participating in the test. (The test was equivalent to running the manual job with just this patch)
Then again this may also one of the usual SD testing race conditions. Since the code in the patch seemed to be related to storage I didn't want to just assume that.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
What's the fix for this for those of us using ovirt-system-tests? Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
I'm not sure it's related since the LSM test runs much later, in 004
On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
Adding Allon & Benny.
Is it possible to verify this is related to the LSM issue we've been handling in [1]? If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.
Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?
[1] https://gerrit.ovirt.org/#/c/78613/
On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com>
Test failed: [ 002_bootstrap.add_secondary_storage_domains ]
Link to suspected patches: https://gerrit.ovirt.org/#/c/79974
This patch adds missing log when resizing an online disk - why do you
wrote: think
it is related to the failure?
Because it is the only patch participating in the test. (The test was equivalent to running the manual job with just this patch)
Then again this may also one of the usual SD testing race conditions. Since the code in the patch seemed to be related to storage I didn't want to just assume that.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
What's the fix for this for those of us using ovirt-system-tests?
Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap
It fails consistently? AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage?
On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
I'm not sure it's related since the LSM test runs much later, in 004
On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
Adding Allon & Benny.
Is it possible to verify this is related to the LSM issue we've been handling in [1]? If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.
Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?
[1] https://gerrit.ovirt.org/#/c/78613/
On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com>
Test failed: [ 002_bootstrap.add_secondary_storage_domains ]
Link to suspected patches: https://gerrit.ovirt.org/#/c/79974
This patch adds missing log when resizing an online disk - why do you
wrote: think
it is related to the failure?
Because it is the only patch participating in the test. (The test was equivalent to running the manual job with just this patch)
Then again this may also one of the usual SD testing race conditions. Since the code in the patch seemed to be related to storage I didn't want to just assume that.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal edri ASSOCIATE MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Edit: reply-all It does, every time. I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite- master/15/consoleFull Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15 On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:
On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
What's the fix for this for those of us using ovirt-system-tests?
Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap
It fails consistently? AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage?
On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
I'm not sure it's related since the LSM test runs much later, in 004
On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
Adding Allon & Benny.
Is it possible to verify this is related to the LSM issue we've been handling in [1]? If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.
Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?
[1] https://gerrit.ovirt.org/#/c/78613/
On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com>
wrote:
> > Test failed: [ 002_bootstrap.add_secondary_storage_domains ] > > Link to suspected patches: > https://gerrit.ovirt.org/#/c/79974
This patch adds missing log when resizing an online disk - why do you think it is related to the failure?
Because it is the only patch participating in the test. (The test was equivalent to running the manual job with just this patch)
Then again this may also one of the usual SD testing race conditions. Since the code in the patch seemed to be related to storage I didn't want to just assume that.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
Actually I spoke too soon, still fails: + lago ovirt runtest /home/myoung/repos/github/ovirt-system-tests/vagrant/test-scenarios/002_bootstrap.py @ Run test: 002_bootstrap.py: nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] # print_api_ver: # print_api_ver: Success (in 0:00:00) # add_dc: # add_dc: Success (in 0:00:43) # add_cluster: # add_cluster: Success (in 0:00:03) # add_hosts: dd * Collect artifacts: * Collect artifacts: Success (in 0:01:14) # add_hosts: Success (in 0:16:36) # Results located at /home/myoung/repos/github/ovirt-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: Success (in 0:17:26) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Edit: reply-all
It does, every time. I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas ter/15/consoleFull
Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15
On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:
On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
What's the fix for this for those of us using ovirt-system-tests?
Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap
It fails consistently? AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage?
On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
I'm not sure it's related since the LSM test runs much later, in 004
On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
Adding Allon & Benny.
Is it possible to verify this is related to the LSM issue we've been handling in [1]? If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.
Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?
[1] https://gerrit.ovirt.org/#/c/78613/
On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote: > > > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> wrote: >> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] >> >> Link to suspected patches: >> https://gerrit.ovirt.org/#/c/79974 > > > This patch adds missing log when resizing an online disk - why do you think > it is related to the failure?
Because it is the only patch participating in the test. (The test was equivalent to running the manual job with just this patch)
Then again this may also one of the usual SD testing race conditions. Since the code in the patch seemed to be related to storage I didn't want to just assume that.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run. I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally. On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
Actually I spoke too soon, still fails:
+ lago ovirt runtest /home/myoung/repos/github/ovirt-system-tests/vagrant/ test-scenarios/002_bootstrap.py @ Run test: 002_bootstrap.py: nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] # print_api_ver: # print_api_ver: Success (in 0:00:00) # add_dc: # add_dc: Success (in 0:00:43) # add_cluster: # add_cluster: Success (in 0:00:03) # add_hosts: dd * Collect artifacts: * Collect artifacts: Success (in 0:01:14) # add_hosts: Success (in 0:16:36) # Results located at /home/myoung/repos/github/ ovirt-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: Success (in 0:17:26) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed
On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Edit: reply-all
It does, every time. I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas ter/15/consoleFull
Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15
On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:
On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
What's the fix for this for those of us using ovirt-system-tests?
Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap
It fails consistently? AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage?
On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
I'm not sure it's related since the LSM test runs much later, in 004
On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
Adding Allon & Benny.
Is it possible to verify this is related to the LSM issue we've been handling in [1]? If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.
Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?
[1] https://gerrit.ovirt.org/#/c/78613/
On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote: > > > > > > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> > wrote: > >> > >> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] > >> > >> Link to suspected patches: > >> https://gerrit.ovirt.org/#/c/79974 > > > > > > This patch adds missing log when resizing an online disk - why do > you think > > it is related to the failure? > > Because it is the only patch participating in the test. > (The test was equivalent to running the manual job with just this > patch) > > Then again this may also one of the usual SD testing race conditions. > Since the code in the patch seemed to be related to storage I didn't > want to just assume that. > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > > >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests The errors have all been from master branch on the basic suite for 4.1 via: $ ./run_suite.sh basic-suite-4.1/ On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.
I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.
On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
Actually I spoke too soon, still fails:
+ lago ovirt runtest /home/myoung/repos/github/ovir t-system-tests/vagrant/test-scenarios/002_bootstrap.py @ Run test: 002_bootstrap.py: nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] # print_api_ver: # print_api_ver: Success (in 0:00:00) # add_dc: # add_dc: Success (in 0:00:43) # add_cluster: # add_cluster: Success (in 0:00:03) # add_hosts: dd * Collect artifacts: * Collect artifacts: Success (in 0:01:14) # add_hosts: Success (in 0:16:36) # Results located at /home/myoung/repos/github/ovir t-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: Success (in 0:17:26) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed
On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Edit: reply-all
It does, every time. I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas ter/15/consoleFull
Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15
On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:
On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
What's the fix for this for those of us using ovirt-system-tests?
Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap
It fails consistently? AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage?
On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
I'm not sure it's related since the LSM test runs much later, in 004
On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
> Adding Allon & Benny. > > Is it possible to verify this is related to the LSM issue we've been > handling in [1]? > If this is the case, we agreed to disable the test next time it > fails, as the current workarounds with sleep isn't enough. > > Can you confirm this is the case, and so we'll have to disable this > test until one of the RFEs described in [1] is merged? > > > [1] https://gerrit.ovirt.org/#/c/78613/ > > On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> > wrote: > >> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote: >> > >> > >> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> >> wrote: >> >> >> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] >> >> >> >> Link to suspected patches: >> >> https://gerrit.ovirt.org/#/c/79974 >> > >> > >> > This patch adds missing log when resizing an online disk - why do >> you think >> > it is related to the failure? >> >> Because it is the only patch participating in the test. >> (The test was equivalent to running the manual job with just this >> patch) >> >> Then again this may also one of the usual SD testing race >> conditions. >> Since the code in the patch seemed to be related to storage I didn't >> want to just assume that. >> >> >> -- >> Barak Korren >> RHV DevOps team , RHCE, RHCi >> Red Hat EMEA >> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> >> >> > > > -- > > Eyal edri > > > ASSOCIATE MANAGER > > RHV DevOps > > EMEA VIRTUALIZATION R&D > > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. > <https://redhat.com/trusted> > phone: +972-9-7692018 <+972%209-769-2018> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests
I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates.
The errors have all been from master branch on the basic suite for 4.1 via:
$ ./run_suite.sh basic-suite-4.1/
OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible. BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code. There is a section on it on the OST readthedocs page. [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_...
On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.
I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.
On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
Actually I spoke too soon, still fails:
+ lago ovirt runtest /home/myoung/repos/github/ovir t-system-tests/vagrant/test-scenarios/002_bootstrap.py @ Run test: 002_bootstrap.py: nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] # print_api_ver: # print_api_ver: Success (in 0:00:00) # add_dc: # add_dc: Success (in 0:00:43) # add_cluster: # add_cluster: Success (in 0:00:03) # add_hosts: dd * Collect artifacts: * Collect artifacts: Success (in 0:01:14) # add_hosts: Success (in 0:16:36) # Results located at /home/myoung/repos/github/ovir t-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: Success (in 0:17:26) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed
On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Edit: reply-all
It does, every time. I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas ter/15/consoleFull
Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15
On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:
On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
What's the fix for this for those of us using ovirt-system-tests?
Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap
It fails consistently? AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage?
On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
> I'm not sure it's related since the LSM test runs much later, in 004 > > On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote: > >> Adding Allon & Benny. >> >> Is it possible to verify this is related to the LSM issue we've >> been handling in [1]? >> If this is the case, we agreed to disable the test next time it >> fails, as the current workarounds with sleep isn't enough. >> >> Can you confirm this is the case, and so we'll have to disable this >> test until one of the RFEs described in [1] is merged? >> >> >> [1] https://gerrit.ovirt.org/#/c/78613/ >> >> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> >> wrote: >> >>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote: >>> > >>> > >>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> >>> wrote: >>> >> >>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] >>> >> >>> >> Link to suspected patches: >>> >> https://gerrit.ovirt.org/#/c/79974 >>> > >>> > >>> > This patch adds missing log when resizing an online disk - why >>> do you think >>> > it is related to the failure? >>> >>> Because it is the only patch participating in the test. >>> (The test was equivalent to running the manual job with just this >>> patch) >>> >>> Then again this may also one of the usual SD testing race >>> conditions. >>> Since the code in the patch seemed to be related to storage I >>> didn't >>> want to just assume that. >>> >>> >>> -- >>> Barak Korren >>> RHV DevOps team , RHCE, RHCi >>> Red Hat EMEA >>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>> _______________________________________________ >>> Devel mailing list >>> Devel@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >>> >>> >> >> >> -- >> >> Eyal edri >> >> >> ASSOCIATE MANAGER >> >> RHV DevOps >> >> EMEA VIRTUALIZATION R&D >> >> >> Red Hat EMEA <https://www.redhat.com/> >> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >> <https://redhat.com/trusted> >> phone: +972-9-7692018 <+972%209-769-2018> >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > > > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
-- Eyal edri ASSOCIATE MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup). stdout logs from a run: https://pastebin.com/KBDaCCYp On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests
I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates.
The errors have all been from master branch on the basic suite for 4.1 via:
$ ./run_suite.sh basic-suite-4.1/
OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.
BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code. There is a section on it on the OST readthedocs page.
[1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/ job/ovirt-system-tests_manual/
On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.
I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.
On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
Actually I spoke too soon, still fails:
+ lago ovirt runtest /home/myoung/repos/github/ovir t-system-tests/vagrant/test-scenarios/002_bootstrap.py @ Run test: 002_bootstrap.py: nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] # print_api_ver: # print_api_ver: Success (in 0:00:00) # add_dc: # add_dc: Success (in 0:00:43) # add_cluster: # add_cluster: Success (in 0:00:03) # add_hosts: dd * Collect artifacts: * Collect artifacts: Success (in 0:01:14) # add_hosts: Success (in 0:16:36) # Results located at /home/myoung/repos/github/ovir t-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: Success (in 0:17:26) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed
On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Edit: reply-all
It does, every time. I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas ter/15/consoleFull
Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15
On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:
On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
> What's the fix for this for those of us using ovirt-system-tests? > > Im trying to adapt some of the code for testing a third party tool, > but master is still failing on 002_bootstrap >
It fails consistently? AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage?
> > On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> > wrote: > >> I'm not sure it's related since the LSM test runs much later, in 004 >> >> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote: >> >>> Adding Allon & Benny. >>> >>> Is it possible to verify this is related to the LSM issue we've >>> been handling in [1]? >>> If this is the case, we agreed to disable the test next time it >>> fails, as the current workarounds with sleep isn't enough. >>> >>> Can you confirm this is the case, and so we'll have to disable >>> this test until one of the RFEs described in [1] is merged? >>> >>> >>> [1] https://gerrit.ovirt.org/#/c/78613/ >>> >>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> >>> wrote: >>> >>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote: >>>> > >>>> > >>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> >>>> wrote: >>>> >> >>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] >>>> >> >>>> >> Link to suspected patches: >>>> >> https://gerrit.ovirt.org/#/c/79974 >>>> > >>>> > >>>> > This patch adds missing log when resizing an online disk - why >>>> do you think >>>> > it is related to the failure? >>>> >>>> Because it is the only patch participating in the test. >>>> (The test was equivalent to running the manual job with just this >>>> patch) >>>> >>>> Then again this may also one of the usual SD testing race >>>> conditions. >>>> Since the code in the patch seemed to be related to storage I >>>> didn't >>>> want to just assume that. >>>> >>>> >>>> -- >>>> Barak Korren >>>> RHV DevOps team , RHCE, RHCi >>>> Red Hat EMEA >>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>>> >>>> >>> >>> >>> -- >>> >>> Eyal edri >>> >>> >>> ASSOCIATE MANAGER >>> >>> RHV DevOps >>> >>> EMEA VIRTUALIZATION R&D >>> >>> >>> Red Hat EMEA <https://www.redhat.com/> >>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>> <https://redhat.com/trusted> >>> phone: +972-9-7692018 <+972%209-769-2018> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >> >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).
stdout logs from a run: https://pastebin.com/KBDaCCYp
Can you send engine log so we'll understand why the host installation failed? You should have all logs needed @/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago_logs Y.
On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests
I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates.
The errors have all been from master branch on the basic suite for 4.1 via:
$ ./run_suite.sh basic-suite-4.1/
OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.
BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code. There is a section on it on the OST readthedocs page.
[1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job /ovirt-system-tests_manual/
On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.
I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.
On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
Actually I spoke too soon, still fails:
+ lago ovirt runtest /home/myoung/repos/github/ovir t-system-tests/vagrant/test-scenarios/002_bootstrap.py @ Run test: 002_bootstrap.py: nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] # print_api_ver: # print_api_ver: Success (in 0:00:00) # add_dc: # add_dc: Success (in 0:00:43) # add_cluster: # add_cluster: Success (in 0:00:03) # add_hosts: dd * Collect artifacts: * Collect artifacts: Success (in 0:01:14) # add_hosts: Success (in 0:16:36) # Results located at /home/myoung/repos/github/ovir t-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: Success (in 0:17:26) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed
On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Edit: reply-all
It does, every time. I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas ter/15/consoleFull
Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15
On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:
> > > On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> > wrote: > >> What's the fix for this for those of us using ovirt-system-tests? >> >> Im trying to adapt some of the code for testing a third party tool, >> but master is still failing on 002_bootstrap >> > > It fails consistently? > AFAIK this is a race condition that happens maybe once a week, can > you share your logs? is it also failing on add_secondary_storage? > > > >> >> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> >> wrote: >> >>> I'm not sure it's related since the LSM test runs much later, in >>> 004 >>> >>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> >>> wrote: >>> >>>> Adding Allon & Benny. >>>> >>>> Is it possible to verify this is related to the LSM issue we've >>>> been handling in [1]? >>>> If this is the case, we agreed to disable the test next time it >>>> fails, as the current workarounds with sleep isn't enough. >>>> >>>> Can you confirm this is the case, and so we'll have to disable >>>> this test until one of the RFEs described in [1] is merged? >>>> >>>> >>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>> >>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> >>>> wrote: >>>> >>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>> wrote: >>>>> > >>>>> > >>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>> bkorren@redhat.com> wrote: >>>>> >> >>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] >>>>> >> >>>>> >> Link to suspected patches: >>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>> > >>>>> > >>>>> > This patch adds missing log when resizing an online disk - why >>>>> do you think >>>>> > it is related to the failure? >>>>> >>>>> Because it is the only patch participating in the test. >>>>> (The test was equivalent to running the manual job with just >>>>> this patch) >>>>> >>>>> Then again this may also one of the usual SD testing race >>>>> conditions. >>>>> Since the code in the patch seemed to be related to storage I >>>>> didn't >>>>> want to just assume that. >>>>> >>>>> >>>>> -- >>>>> Barak Korren >>>>> RHV DevOps team , RHCE, RHCi >>>>> Red Hat EMEA >>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Eyal edri >>>> >>>> >>>> ASSOCIATE MANAGER >>>> >>>> RHV DevOps >>>> >>>> EMEA VIRTUALIZATION R&D >>>> >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>> <https://redhat.com/trusted> >>>> phone: +972-9-7692018 <+972%209-769-2018> >>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> > > > -- > > Eyal edri > > > ASSOCIATE MANAGER > > RHV DevOps > > EMEA VIRTUALIZATION R&D > > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. > <https://redhat.com/trusted> > phone: +972-9-7692018 <+972%209-769-2018> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Full log is just as uninteresting: https://paste.fedoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).
stdout logs from a run: https://pastebin.com/KBDaCCYp
Can you send engine log so we'll understand why the host installation failed? You should have all logs needed @/home/myoung/repos/github/ ovirt-system-tests/test_logs/basic-suite-4.1/post-002_ bootstrap.py/lago_logs
Y.
On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests
I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates.
The errors have all been from master branch on the basic suite for 4.1 via:
$ ./run_suite.sh basic-suite-4.1/
OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.
BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code. There is a section on it on the OST readthedocs page.
[1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job /ovirt-system-tests_manual/
On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.
I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.
On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
Actually I spoke too soon, still fails:
+ lago ovirt runtest /home/myoung/repos/github/ovir t-system-tests/vagrant/test-scenarios/002_bootstrap.py @ Run test: 002_bootstrap.py: nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$'] # print_api_ver: # print_api_ver: Success (in 0:00:00) # add_dc: # add_dc: Success (in 0:00:43) # add_cluster: # add_cluster: Success (in 0:00:03) # add_hosts: dd * Collect artifacts: * Collect artifacts: Success (in 0:01:14) # add_hosts: Success (in 0:16:36) # Results located at /home/myoung/repos/github/ovir t-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: Success (in 0:17:26) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed
On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
> Edit: reply-all > > It does, every time. > I got it to pass by using the last passing revision from Jenkins > (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from > http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas > ter/15/consoleFull > > Not sure if it's a true race condition, that build in Jenkins has > failed consistently since #15 > > On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote: > >> >> >> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> >> wrote: >> >>> What's the fix for this for those of us using ovirt-system-tests? >>> >>> Im trying to adapt some of the code for testing a third party >>> tool, but master is still failing on 002_bootstrap >>> >> >> It fails consistently? >> AFAIK this is a race condition that happens maybe once a week, can >> you share your logs? is it also failing on add_secondary_storage? >> >> >> >>> >>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com >>> > wrote: >>> >>>> I'm not sure it's related since the LSM test runs much later, in >>>> 004 >>>> >>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> >>>> wrote: >>>> >>>>> Adding Allon & Benny. >>>>> >>>>> Is it possible to verify this is related to the LSM issue we've >>>>> been handling in [1]? >>>>> If this is the case, we agreed to disable the test next time it >>>>> fails, as the current workarounds with sleep isn't enough. >>>>> >>>>> Can you confirm this is the case, and so we'll have to disable >>>>> this test until one of the RFEs described in [1] is merged? >>>>> >>>>> >>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>> >>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com >>>>> > wrote: >>>>> >>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>>> wrote: >>>>>> > >>>>>> > >>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>> bkorren@redhat.com> wrote: >>>>>> >> >>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains ] >>>>>> >> >>>>>> >> Link to suspected patches: >>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>> > >>>>>> > >>>>>> > This patch adds missing log when resizing an online disk - >>>>>> why do you think >>>>>> > it is related to the failure? >>>>>> >>>>>> Because it is the only patch participating in the test. >>>>>> (The test was equivalent to running the manual job with just >>>>>> this patch) >>>>>> >>>>>> Then again this may also one of the usual SD testing race >>>>>> conditions. >>>>>> Since the code in the patch seemed to be related to storage I >>>>>> didn't >>>>>> want to just assume that. >>>>>> >>>>>> >>>>>> -- >>>>>> Barak Korren >>>>>> RHV DevOps team , RHCE, RHCi >>>>>> Red Hat EMEA >>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Eyal edri >>>>> >>>>> >>>>> ASSOCIATE MANAGER >>>>> >>>>> RHV DevOps >>>>> >>>>> EMEA VIRTUALIZATION R&D >>>>> >>>>> >>>>> Red Hat EMEA <https://www.redhat.com/> >>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>> <https://redhat.com/trusted> >>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >> >> >> -- >> >> Eyal edri >> >> >> ASSOCIATE MANAGER >> >> RHV DevOps >> >> EMEA VIRTUALIZATION R&D >> >> >> Red Hat EMEA <https://www.redhat.com/> >> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >> <https://redhat.com/trusted> >> phone: +972-9-7692018 <+972%209-769-2018> >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Full log is just as uninteresting: https://paste.fedoraproject.org/paste/ ZaZMcT5sg9SAGQ7OKC3-EA
I was actually hoping for the engine log. Y.
On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).
stdout logs from a run: https://pastebin.com/KBDaCCYp
Can you send engine log so we'll understand why the host installation failed? You should have all logs needed @/home/myoung/repos/github/ovi rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs
Y.
On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests
I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates.
The errors have all been from master branch on the basic suite for 4.1 via:
$ ./run_suite.sh basic-suite-4.1/
OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.
BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code. There is a section on it on the OST readthedocs page.
[1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job /ovirt-system-tests_manual/
On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.
I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.
On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
> Actually I spoke too soon, still fails: > > + lago ovirt runtest /home/myoung/repos/github/ovir > t-system-tests/vagrant/test-scenarios/002_bootstrap.py > @ Run test: 002_bootstrap.py: > nose.config: INFO: Ignoring files matching ['^\\.', '^_', > '^setup\\.py$'] > # print_api_ver: > # print_api_ver: Success (in 0:00:00) > # add_dc: > # add_dc: Success (in 0:00:43) > # add_cluster: > # add_cluster: Success (in 0:00:03) > # add_hosts: > dd > * Collect artifacts: > * Collect artifacts: Success (in 0:01:14) > # add_hosts: Success (in 0:16:36) > # Results located at /home/myoung/repos/github/ovir > t-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml > @ Run test: 002_bootstrap.py: Success (in 0:17:26) > Error occured, aborting > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line > 360, in do_run > self.cli_plugins[args.ovirtverb].do_run(args) > File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line > 184, in do_run > self._do_run(**vars(args)) > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, > in wrapper > return func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, > in wrapper > return func(*args, prefix=prefix, **kwargs) > File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line > 99, in do_ovirt_runtest > raise RuntimeError('Some tests failed') > RuntimeError: Some tests failed > > > > > On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> > wrote: > >> Edit: reply-all >> >> It does, every time. >> I got it to pass by using the last passing revision from Jenkins >> (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from >> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas >> ter/15/consoleFull >> >> Not sure if it's a true race condition, that build in Jenkins has >> failed consistently since #15 >> >> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> >> wrote: >> >>> >>> >>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com >>> > wrote: >>> >>>> What's the fix for this for those of us using ovirt-system-tests? >>>> >>>> Im trying to adapt some of the code for testing a third party >>>> tool, but master is still failing on 002_bootstrap >>>> >>> >>> It fails consistently? >>> AFAIK this is a race condition that happens maybe once a week, can >>> you share your logs? is it also failing on add_secondary_storage? >>> >>> >>> >>>> >>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>> bzlotnik@redhat.com> wrote: >>>> >>>>> I'm not sure it's related since the LSM test runs much later, in >>>>> 004 >>>>> >>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> >>>>> wrote: >>>>> >>>>>> Adding Allon & Benny. >>>>>> >>>>>> Is it possible to verify this is related to the LSM issue we've >>>>>> been handling in [1]? >>>>>> If this is the case, we agreed to disable the test next time it >>>>>> fails, as the current workarounds with sleep isn't enough. >>>>>> >>>>>> Can you confirm this is the case, and so we'll have to disable >>>>>> this test until one of the RFEs described in [1] is merged? >>>>>> >>>>>> >>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>> >>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>> bkorren@redhat.com> wrote: >>>>>> >>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > >>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>> bkorren@redhat.com> wrote: >>>>>>> >> >>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>> ] >>>>>>> >> >>>>>>> >> Link to suspected patches: >>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>> > >>>>>>> > >>>>>>> > This patch adds missing log when resizing an online disk - >>>>>>> why do you think >>>>>>> > it is related to the failure? >>>>>>> >>>>>>> Because it is the only patch participating in the test. >>>>>>> (The test was equivalent to running the manual job with just >>>>>>> this patch) >>>>>>> >>>>>>> Then again this may also one of the usual SD testing race >>>>>>> conditions. >>>>>>> Since the code in the patch seemed to be related to storage I >>>>>>> didn't >>>>>>> want to just assume that. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Barak Korren >>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>> Red Hat EMEA >>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Eyal edri >>>>>> >>>>>> >>>>>> ASSOCIATE MANAGER >>>>>> >>>>>> RHV DevOps >>>>>> >>>>>> EMEA VIRTUALIZATION R&D >>>>>> >>>>>> >>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>> <https://redhat.com/trusted> >>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel@ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> Eyal edri >>> >>> >>> ASSOCIATE MANAGER >>> >>> RHV DevOps >>> >>> EMEA VIRTUALIZATION R&D >>> >>> >>> Red Hat EMEA <https://www.redhat.com/> >>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>> <https://redhat.com/trusted> >>> phone: +972-9-7692018 <+972%209-769-2018> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >> >> >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
You should have all logs needed @/home/myoung/repos/github/ ovirt-system-tests/test_logs/basic-suite-4.1/post-002_ bootstrap.py/lago_logs
The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. Heres all logs for basic suite 4.1 that I know how to find: myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; done Uploading (1355.0KiB)... https://da.gd/OIlR -> https://paste.fedoraproject.org/paste/9Xn2RuDgF2pY2I0CouBbGg/ Uploading (1154.0KiB)... https://da.gd/0Cti -> https://paste.fedoraproject.org/paste/uJqvAiDSblnmN8OA6i5tXA/ Uploading (1377.8KiB)... https://da.gd/GpDIf -> https://paste.fedoraproject.org/paste/~Cn6IJgbvateJfFulDNFCg/ Uploading (1176.7KiB)... https://da.gd/skhBy -> https://paste.fedoraproject.org/paste/HIpEVm4TRPpH5nJCOyUfWA/ Uploading (1397.1KiB)... https://da.gd/fEtF -> https://paste.fedoraproject.org/paste/cuycP~u9D~gJKNPsQ5NH2g/ Uploading (1198.1KiB)... https://da.gd/1zNm -> https://paste.fedoraproject.org/paste/5~yEDwCSr~tdxHT~XUaLPg/ On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Full log is just as uninteresting: https://paste.f edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
I was actually hoping for the engine log. Y.
On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).
stdout logs from a run: https://pastebin.com/KBDaCCYp
Can you send engine log so we'll understand why the host installation failed? You should have all logs needed @/home/myoung/repos/github/ovi rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap .py/lago_logs
Y.
On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests
I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates.
The errors have all been from master branch on the basic suite for 4.1 via:
$ ./run_suite.sh basic-suite-4.1/
OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.
BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code. There is a section on it on the OST readthedocs page.
[1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job /ovirt-system-tests_manual/
On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
> I see you run the 'hc' suite, which means hyperconverged. This suite > is running an hosted engine on gluster storage, It's a more complex suite > than the basic one, and prone to more errors, however it should still work, > but if you don't require it specifically, I would recommend running the > basic suite, which should be easier to debug and also quicker to run. > > I'm also adding hc maintainer so she can check, can you share the > link to the Jenkins job you're running? Or the log files if you're running > locally. > > On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote: > >> Actually I spoke too soon, still fails: >> >> + lago ovirt runtest /home/myoung/repos/github/ovir >> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >> @ Run test: 002_bootstrap.py: >> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >> '^setup\\.py$'] >> # print_api_ver: >> # print_api_ver: Success (in 0:00:00) >> # add_dc: >> # add_dc: Success (in 0:00:43) >> # add_cluster: >> # add_cluster: Success (in 0:00:03) >> # add_hosts: >> dd >> * Collect artifacts: >> * Collect artifacts: Success (in 0:01:14) >> # add_hosts: Success (in 0:16:36) >> # Results located at /home/myoung/repos/github/ovir >> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j >> unit.xml >> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >> Error occured, aborting >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >> 360, in do_run >> self.cli_plugins[args.ovirtverb].do_run(args) >> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >> line 184, in do_run >> self._do_run(**vars(args)) >> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, >> in wrapper >> return func(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, >> in wrapper >> return func(*args, prefix=prefix, **kwargs) >> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >> 99, in do_ovirt_runtest >> raise RuntimeError('Some tests failed') >> RuntimeError: Some tests failed >> >> >> >> >> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> >> wrote: >> >>> Edit: reply-all >>> >>> It does, every time. >>> I got it to pass by using the last passing revision from Jenkins >>> (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from >>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas >>> ter/15/consoleFull >>> >>> Not sure if it's a true race condition, that build in Jenkins has >>> failed consistently since #15 >>> >>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> >>> wrote: >>> >>>> >>>> >>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young < >>>> 3vilpenguin@gmail.com> wrote: >>>> >>>>> What's the fix for this for those of us using ovirt-system-tests? >>>>> >>>>> Im trying to adapt some of the code for testing a third party >>>>> tool, but master is still failing on 002_bootstrap >>>>> >>>> >>>> It fails consistently? >>>> AFAIK this is a race condition that happens maybe once a week, >>>> can you share your logs? is it also failing on add_secondary_storage? >>>> >>>> >>>> >>>>> >>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>>> bzlotnik@redhat.com> wrote: >>>>> >>>>>> I'm not sure it's related since the LSM test runs much later, >>>>>> in 004 >>>>>> >>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> >>>>>> wrote: >>>>>> >>>>>>> Adding Allon & Benny. >>>>>>> >>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>> we've been handling in [1]? >>>>>>> If this is the case, we agreed to disable the test next time >>>>>>> it fails, as the current workarounds with sleep isn't enough. >>>>>>> >>>>>>> Can you confirm this is the case, and so we'll have to disable >>>>>>> this test until one of the RFEs described in [1] is merged? >>>>>>> >>>>>>> >>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>> >>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>>> bkorren@redhat.com> wrote: >>>>>>> >>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>>>>> wrote: >>>>>>>> > >>>>>>>> > >>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>>> bkorren@redhat.com> wrote: >>>>>>>> >> >>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>>> ] >>>>>>>> >> >>>>>>>> >> Link to suspected patches: >>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>> > >>>>>>>> > >>>>>>>> > This patch adds missing log when resizing an online disk - >>>>>>>> why do you think >>>>>>>> > it is related to the failure? >>>>>>>> >>>>>>>> Because it is the only patch participating in the test. >>>>>>>> (The test was equivalent to running the manual job with just >>>>>>>> this patch) >>>>>>>> >>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>> conditions. >>>>>>>> Since the code in the patch seemed to be related to storage I >>>>>>>> didn't >>>>>>>> want to just assume that. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Barak Korren >>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>> Red Hat EMEA >>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list >>>>>>>> Devel@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Eyal edri >>>>>>> >>>>>>> >>>>>>> ASSOCIATE MANAGER >>>>>>> >>>>>>> RHV DevOps >>>>>>> >>>>>>> EMEA VIRTUALIZATION R&D >>>>>>> >>>>>>> >>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>> <https://redhat.com/trusted> >>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Devel mailing list >>>>>> Devel@ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Eyal edri >>>> >>>> >>>> ASSOCIATE MANAGER >>>> >>>> RHV DevOps >>>> >>>> EMEA VIRTUALIZATION R&D >>>> >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>> <https://redhat.com/trusted> >>>> phone: +972-9-7692018 <+972%209-769-2018> >>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >>> >>> >>
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin@gmail.com> wrote:
You should have all logs needed @/home/myoung/repos/github/ovi
rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs
The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. Heres all logs for basic suite 4.1 that I know how to find:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
The engine and hosts logs are usually under the test_logs, which should be in the same subdir as lago_logs directory. Also, can you verify you have the latest python v4 SDK installled? rpm -q python-ovirt-engine-sdk4 If you don't have it installed, you can get it directly from the build job ( which acts also as yum repo ): el7 build - http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_4.1_build-artifacts-el... fc26 build is available only for master [1] but should also work. There are also fc24/25 builds, just follow the same pattern to find them on Jenkins. [1] http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_master_build-artifacts...
$ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; done Uploading (1355.0KiB)... https://da.gd/OIlR -> https://paste.fedoraproject.org/paste/ 9Xn2RuDgF2pY2I0CouBbGg/ Uploading (1154.0KiB)... https://da.gd/0Cti -> https://paste.fedoraproject.org/paste/ uJqvAiDSblnmN8OA6i5tXA/ Uploading (1377.8KiB)... https://da.gd/GpDIf -> https://paste.fedoraproject.org/paste/~ Cn6IJgbvateJfFulDNFCg/ Uploading (1176.7KiB)... https://da.gd/skhBy -> https://paste.fedoraproject.org/paste/ HIpEVm4TRPpH5nJCOyUfWA/ Uploading (1397.1KiB)... https://da.gd/fEtF -> https://paste.fedoraproject.org/paste/cuycP~u9D~ gJKNPsQ5NH2g/ Uploading (1198.1KiB)... https://da.gd/1zNm -> https://paste.fedoraproject. org/paste/5~yEDwCSr~tdxHT~XUaLPg/
On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Full log is just as uninteresting: https://paste.f edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
I was actually hoping for the engine log. Y.
On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).
stdout logs from a run: https://pastebin.com/KBDaCCYp
Can you send engine log so we'll understand why the host installation failed? You should have all logs needed @/home/myoung/repos/github/ovi rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap .py/lago_logs
Y.
On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
> Ill try to get some detailed log files later, but fwiw I'm not > running the hc suites (afaik, Im still getting inundated with the system > tests/lago). The link I used for Jenkins was just to try to pull the latest > 'passing' hash for ovirt-system-tests >
I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates.
> > The errors have all been from master branch on the basic suite for > 4.1 via: > > $ ./run_suite.sh basic-suite-4.1/ > > OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.
BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code. There is a section on it on the OST readthedocs page.
[1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job /ovirt-system-tests_manual/
> > On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> > wrote: > >> I see you run the 'hc' suite, which means hyperconverged. This >> suite is running an hosted engine on gluster storage, It's a more complex >> suite than the basic one, and prone to more errors, however it should still >> work, but if you don't require it specifically, I would recommend running >> the basic suite, which should be easier to debug and also quicker to run. >> >> I'm also adding hc maintainer so she can check, can you share the >> link to the Jenkins job you're running? Or the log files if you're running >> locally. >> >> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote: >> >>> Actually I spoke too soon, still fails: >>> >>> + lago ovirt runtest /home/myoung/repos/github/ovir >>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >>> @ Run test: 002_bootstrap.py: >>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>> '^setup\\.py$'] >>> # print_api_ver: >>> # print_api_ver: Success (in 0:00:00) >>> # add_dc: >>> # add_dc: Success (in 0:00:43) >>> # add_cluster: >>> # add_cluster: Success (in 0:00:03) >>> # add_hosts: >>> dd >>> * Collect artifacts: >>> * Collect artifacts: Success (in 0:01:14) >>> # add_hosts: Success (in 0:16:36) >>> # Results located at /home/myoung/repos/github/ovir >>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j >>> unit.xml >>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>> Error occured, aborting >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>> 360, in do_run >>> self.cli_plugins[args.ovirtverb].do_run(args) >>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >>> line 184, in do_run >>> self._do_run(**vars(args)) >>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>> 501, in wrapper >>> return func(*args, **kwargs) >>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>> 512, in wrapper >>> return func(*args, prefix=prefix, **kwargs) >>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>> 99, in do_ovirt_runtest >>> raise RuntimeError('Some tests failed') >>> RuntimeError: Some tests failed >>> >>> >>> >>> >>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com >>> > wrote: >>> >>>> Edit: reply-all >>>> >>>> It does, every time. >>>> I got it to pass by using the last passing revision from Jenkins >>>> (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from >>>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas >>>> ter/15/consoleFull >>>> >>>> Not sure if it's a true race condition, that build in Jenkins has >>>> failed consistently since #15 >>>> >>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young < >>>>> 3vilpenguin@gmail.com> wrote: >>>>> >>>>>> What's the fix for this for those of us using >>>>>> ovirt-system-tests? >>>>>> >>>>>> Im trying to adapt some of the code for testing a third party >>>>>> tool, but master is still failing on 002_bootstrap >>>>>> >>>>> >>>>> It fails consistently? >>>>> AFAIK this is a race condition that happens maybe once a week, >>>>> can you share your logs? is it also failing on add_secondary_storage? >>>>> >>>>> >>>>> >>>>>> >>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>>>> bzlotnik@redhat.com> wrote: >>>>>> >>>>>>> I'm not sure it's related since the LSM test runs much later, >>>>>>> in 004 >>>>>>> >>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Adding Allon & Benny. >>>>>>>> >>>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>>> we've been handling in [1]? >>>>>>>> If this is the case, we agreed to disable the test next time >>>>>>>> it fails, as the current workarounds with sleep isn't enough. >>>>>>>> >>>>>>>> Can you confirm this is the case, and so we'll have to >>>>>>>> disable this test until one of the RFEs described in [1] is merged? >>>>>>>> >>>>>>>> >>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>> >>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>>>> bkorren@redhat.com> wrote: >>>>>>>> >>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>>>>>> wrote: >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>> >> >>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>>>> ] >>>>>>>>> >> >>>>>>>>> >> Link to suspected patches: >>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > This patch adds missing log when resizing an online disk - >>>>>>>>> why do you think >>>>>>>>> > it is related to the failure? >>>>>>>>> >>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>> (The test was equivalent to running the manual job with just >>>>>>>>> this patch) >>>>>>>>> >>>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>>> conditions. >>>>>>>>> Since the code in the patch seemed to be related to storage >>>>>>>>> I didn't >>>>>>>>> want to just assume that. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Barak Korren >>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>> Red Hat EMEA >>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list >>>>>>>>> Devel@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Eyal edri >>>>>>>> >>>>>>>> >>>>>>>> ASSOCIATE MANAGER >>>>>>>> >>>>>>>> RHV DevOps >>>>>>>> >>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>> >>>>>>>> >>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>> <https://redhat.com/trusted> >>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list >>>>>>> Devel@ovirt.org >>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Eyal edri >>>>> >>>>> >>>>> ASSOCIATE MANAGER >>>>> >>>>> RHV DevOps >>>>> >>>>> EMEA VIRTUALIZATION R&D >>>>> >>>>> >>>>> Red Hat EMEA <https://www.redhat.com/> >>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>> <https://redhat.com/trusted> >>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>> >>>> >>>> >>> >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
-- Eyal edri ASSOCIATE MANAGER RHV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
rpm -q python-ovirt-engine-sdk4
Mine is outdated: $ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64 Ill update that and run again. Logs (better to send too many than not enough sometimes). Ovirt engine logs are there. myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/tuned/tuned.log Uploading (3.5KiB)... https://da.gd/LiG0 -> https://paste.fedoraproject.org/paste/wed7pYARTmHHTHcf0DvVAQ/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/audit/audit.log Uploading (401.2KiB)... https://da.gd/tgSw -> https://paste.fedoraproject.org/paste/WEqegOFiLpiNGCv0hvY0ww/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/yum.log Uploading (24.3KiB)... https://da.gd/BzoMg -> https://paste.fedoraproject.org/paste/qla7z~hRsym8kkCr6NlXFg/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/setup-ds.log Uploading (37.4KiB)... https://da.gd/1HHKA -> https://paste.fedoraproject.org/paste/4nRRDoC8lrv5CTxbS6TCxQ/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20170814074645-vzih31.log Uploading (5544.4KiB)... https://da.gd/xYpYl -> https://paste.fedoraproject.org/paste/goz7RQVvusLXjiiheLD5xg/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.3KiB)... https://da.gd/4Y6pE -> https://paste.fedoraproject.org/paste/OCqWe6tjGkrNOrTkqex3mg/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (34.1KiB)... https://da.gd/PZm7d -> https://paste.fedoraproject.org/paste/HpZOi8LTrgcWhKqfPeowPA/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2724.0KiB)... https://da.gd/pIOO -> https://paste.fedoraproject.org/paste/ufbtnfZOfSmun8sbOCs1xg/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/Xf5R -> https://paste.fedoraproject.org/paste/MO-dlfNQXi19Tla3T~QXsA/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/snmptrapd.log Uploading (0.1KiB)... https://da.gd/cF48 -> https://paste.fedoraproject.org/paste/NYqECCL1QVihcBQV~VWDsQ/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/G5oC -> https://paste.fedoraproject.org/paste/JgvhOLoA7cEO2Q3ESjP5Aw/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/audit/audit.log Uploading (183.1KiB)... https://da.gd/9bjYn -> https://paste.fedoraproject.org/paste/Ado1joqnZxkPL1us6UeOwQ/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/yum.log Uploading (18.3KiB)... https://da.gd/i5ki -> https://paste.fedoraproject.org/paste/9sxQziGLRqw1tLHKfBAi6Q/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/5Iiu4 -> https://paste.fedoraproject.org/paste/RaqsqKQdfgcTP1o2Epo~Pg/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/audit/audit.log Uploading (179.9KiB)... https://da.gd/MyZ7 -> https://paste.fedoraproject.org/paste/x7T7ZkIS3g5JE2dj028AVg/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/yum.log Uploading (17.4KiB)... https://da.gd/Gmux -> https://paste.fedoraproject.org/paste/kORmR4ShCv1EVGtTWSu2Zw/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs/lago.log Uploading (1198.1KiB)... https://da.gd/vzVCA -> https://paste.fedoraproject.org/paste/TuQiMEro23I~OnFcM7wG-g/ ---- On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin@gmail.com> wrote:
You should have all logs needed @/home/myoung/repos/github/ovi
rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap .py/lago_logs
The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. Heres all logs for basic suite 4.1 that I know how to find:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
The engine and hosts logs are usually under the test_logs, which should be in the same subdir as lago_logs directory. Also, can you verify you have the latest python v4 SDK installled?
rpm -q python-ovirt-engine-sdk4
If you don't have it installed, you can get it directly from the build job ( which acts also as yum repo ):
el7 build - http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_4.1_ build-artifacts-el7-x86_64/lastSuccessfulBuild/artifact/ exported-artifacts/
fc26 build is available only for master [1] but should also work.
There are also fc24/25 builds, just follow the same pattern to find them on Jenkins.
[1] http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_ master_build-artifacts-fc26-x86_64/lastSuccessfulBuild/ artifact/exported-artifacts/
$ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; done Uploading (1355.0KiB)... https://da.gd/OIlR -> https://paste.fedoraproject.or g/paste/9Xn2RuDgF2pY2I0CouBbGg/ Uploading (1154.0KiB)... https://da.gd/0Cti -> https://paste.fedoraproject.or g/paste/uJqvAiDSblnmN8OA6i5tXA/ Uploading (1377.8KiB)... https://da.gd/GpDIf -> https://paste.fedoraproject.or g/paste/~Cn6IJgbvateJfFulDNFCg/ Uploading (1176.7KiB)... https://da.gd/skhBy -> https://paste.fedoraproject.or g/paste/HIpEVm4TRPpH5nJCOyUfWA/ Uploading (1397.1KiB)... https://da.gd/fEtF -> https://paste.fedoraproject.or g/paste/cuycP~u9D~gJKNPsQ5NH2g/ Uploading (1198.1KiB)... https://da.gd/1zNm -> https://paste.fedoraproject.or g/paste/5~yEDwCSr~tdxHT~XUaLPg/
On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Full log is just as uninteresting: https://paste.f edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
I was actually hoping for the engine log. Y.
On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).
stdout logs from a run: https://pastebin.com/KBDaCCYp
Can you send engine log so we'll understand why the host installation failed? You should have all logs needed @/home/myoung/repos/github/ovi rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap .py/lago_logs
Y.
On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:
> > > On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> > wrote: > >> Ill try to get some detailed log files later, but fwiw I'm not >> running the hc suites (afaik, Im still getting inundated with the system >> tests/lago). The link I used for Jenkins was just to try to pull the latest >> 'passing' hash for ovirt-system-tests >> > > I see, its a bit more complicated than that I'm afraid, the HASH you > see is probably on for OST code itself, while the suites can fail on any > oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its > more than just a single hash, but rather a list of RPMs and their versions, > together with hash of OST ( for the tests themselves ) and also OS updates. > > >> >> The errors have all been from master branch on the basic suite for >> 4.1 via: >> >> $ ./run_suite.sh basic-suite-4.1/ >> >> > OK, then we need to investigate it and understand why add host fails > on 4.1, please provide logs when possible. > > BTW, you can also try running the manual job [1], but you'll need to > provide it with custom yum repo URL with your built artifacts to test your > code. > There is a section on it on the OST readthedocs page. > > [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job > /ovirt-system-tests_manual/ > > >> >> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> >> wrote: >> >>> I see you run the 'hc' suite, which means hyperconverged. This >>> suite is running an hosted engine on gluster storage, It's a more complex >>> suite than the basic one, and prone to more errors, however it should still >>> work, but if you don't require it specifically, I would recommend running >>> the basic suite, which should be easier to debug and also quicker to run. >>> >>> I'm also adding hc maintainer so she can check, can you share the >>> link to the Jenkins job you're running? Or the log files if you're running >>> locally. >>> >>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote: >>> >>>> Actually I spoke too soon, still fails: >>>> >>>> + lago ovirt runtest /home/myoung/repos/github/ovir >>>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >>>> @ Run test: 002_bootstrap.py: >>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>>> '^setup\\.py$'] >>>> # print_api_ver: >>>> # print_api_ver: Success (in 0:00:00) >>>> # add_dc: >>>> # add_dc: Success (in 0:00:43) >>>> # add_cluster: >>>> # add_cluster: Success (in 0:00:03) >>>> # add_hosts: >>>> dd >>>> * Collect artifacts: >>>> * Collect artifacts: Success (in 0:01:14) >>>> # add_hosts: Success (in 0:16:36) >>>> # Results located at /home/myoung/repos/github/ovir >>>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j >>>> unit.xml >>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>>> Error occured, aborting >>>> Traceback (most recent call last): >>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>>> 360, in do_run >>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >>>> line 184, in do_run >>>> self._do_run(**vars(args)) >>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>> 501, in wrapper >>>> return func(*args, **kwargs) >>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>> 512, in wrapper >>>> return func(*args, prefix=prefix, **kwargs) >>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>>> 99, in do_ovirt_runtest >>>> raise RuntimeError('Some tests failed') >>>> RuntimeError: Some tests failed >>>> >>>> >>>> >>>> >>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young < >>>> 3vilpenguin@gmail.com> wrote: >>>> >>>>> Edit: reply-all >>>>> >>>>> It does, every time. >>>>> I got it to pass by using the last passing revision from Jenkins >>>>> (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from >>>>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-mas >>>>> ter/15/consoleFull >>>>> >>>>> Not sure if it's a true race condition, that build in Jenkins >>>>> has failed consistently since #15 >>>>> >>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young < >>>>>> 3vilpenguin@gmail.com> wrote: >>>>>> >>>>>>> What's the fix for this for those of us using >>>>>>> ovirt-system-tests? >>>>>>> >>>>>>> Im trying to adapt some of the code for testing a third party >>>>>>> tool, but master is still failing on 002_bootstrap >>>>>>> >>>>>> >>>>>> It fails consistently? >>>>>> AFAIK this is a race condition that happens maybe once a week, >>>>>> can you share your logs? is it also failing on add_secondary_storage? >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>>>>> bzlotnik@redhat.com> wrote: >>>>>>> >>>>>>>> I'm not sure it's related since the LSM test runs much later, >>>>>>>> in 004 >>>>>>>> >>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Adding Allon & Benny. >>>>>>>>> >>>>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>>>> we've been handling in [1]? >>>>>>>>> If this is the case, we agreed to disable the test next time >>>>>>>>> it fails, as the current workarounds with sleep isn't enough. >>>>>>>>> >>>>>>>>> Can you confirm this is the case, and so we'll have to >>>>>>>>> disable this test until one of the RFEs described in [1] is merged? >>>>>>>>> >>>>>>>>> >>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>>> >>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>> >>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>>> >> >>>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>>>>> ] >>>>>>>>>> >> >>>>>>>>>> >> Link to suspected patches: >>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > This patch adds missing log when resizing an online disk >>>>>>>>>> - why do you think >>>>>>>>>> > it is related to the failure? >>>>>>>>>> >>>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>>> (The test was equivalent to running the manual job with >>>>>>>>>> just this patch) >>>>>>>>>> >>>>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>>>> conditions. >>>>>>>>>> Since the code in the patch seemed to be related to storage >>>>>>>>>> I didn't >>>>>>>>>> want to just assume that. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Barak Korren >>>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>>> Red Hat EMEA >>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list >>>>>>>>>> Devel@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Eyal edri >>>>>>>>> >>>>>>>>> >>>>>>>>> ASSOCIATE MANAGER >>>>>>>>> >>>>>>>>> RHV DevOps >>>>>>>>> >>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>> >>>>>>>>> >>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>> <https://redhat.com/trusted> >>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list >>>>>>>> Devel@ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Eyal edri >>>>>> >>>>>> >>>>>> ASSOCIATE MANAGER >>>>>> >>>>>> RHV DevOps >>>>>> >>>>>> EMEA VIRTUALIZATION R&D >>>>>> >>>>>> >>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>> <https://redhat.com/trusted> >>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>> >>>>> >>>>> >>>> >> > > > -- > > Eyal edri > > > ASSOCIATE MANAGER > > RHV DevOps > > EMEA VIRTUALIZATION R&D > > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> TRIED. TESTED. TRUSTED. > <https://redhat.com/trusted> > phone: +972-9-7692018 <+972%209-769-2018> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
After updating the python sdk: myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.6-2.20170712git1b99f36.el7.centos.x86_64 I get more but different errors[1]. Then the lago log[2] All engine Logs are even further[3] [1] # add_cluster: Success (in 0:00:03) # add_hosts: * Collect artifacts: - [Thread-5] lago-basic-suite-4-1-host1: ERROR (in 0:00:19) - [Thread-4] lago-basic-suite-4-1-engine: ERROR (in 0:00:20) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout * Collect artifacts: ERROR (in 0:00:33) # add_hosts: ERROR (in 0:15:52) # Collect artifacts: * [Thread-19] lago-basic-suite-4-1-engine: * [Thread-20] lago-basic-suite-4-1-host1: * [Thread-21] lago-basic-suite-4-1-host0: * [Thread-21] lago-basic-suite-4-1-host0: Success (in 0:00:03) * [Thread-20] lago-basic-suite-4-1-host1: Success (in 0:00:05) * [Thread-19] lago-basic-suite-4-1-engine: Success (in 0:00:16) # Collect artifacts: Success (in 0:00:16) # Results located at /home/myoung/repos/github/ovirt-system-tests/deployment-basic-suite-4.1/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: ERROR (in 0:16:57) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed + res=1 + cd - /home/myoung/repos/github/ovirt-system-tests + return 1 + failed=true + env_collect /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py + local tests_out_dir=/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py + echo '#########################' ######################### + [[ -e /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1 ]] + cd /home/myoung/repos/github/ovirt-system-tests/deployment-basic-suite-4.1/current + lago collect --output /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py @ Collect artifacts: # [Thread-1] lago-basic-suite-4-1-engine: # [Thread-2] lago-basic-suite-4-1-host1: # [Thread-3] lago-basic-suite-4-1-host0: # [Thread-3] lago-basic-suite-4-1-host0: Success (in 0:00:02) # [Thread-2] lago-basic-suite-4-1-host1: Success (in 0:00:03) # [Thread-1] lago-basic-suite-4-1-engine: Success (in 0:00:06) @ Collect artifacts: Success (in 0:00:07) + cp -a logs /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago_logs + cd - /home/myoung/repos/github/ovirt-system-tests + true + echo '@@@@ ERROR: Failed running /home/myoung/repos/github/ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py' @@@@ ERROR: Failed running /home/myoung/repos/github/ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py + return 1 [2] $ cat './test_logs/basic-suite-4.1/post-002_bootstrap.py.2017-08-14_21:55:00/lago_logs/lago.log' | fpaste Uploading (1202.8KiB)... https://da.gd/6aw2 -> https://paste.fedoraproject.org/paste/956xJBRY-m1D9T7giQktHQ/ [3] $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log | grep ovirt-engine); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20170814212716-ql8ekq.log Uploading (5544.7KiB)... https://da.gd/QLkSD -> https://paste.fedoraproject.org/paste/tQEEAfL3TQk7nYhrdyQIVQ/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.5KiB)... https://da.gd/eX4c6 -> https://paste.fedoraproject.org/paste/7YFWv6uVzARJ8q0iCAhUNQ/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (33.9KiB)... https://da.gd/n1JTd -> https://paste.fedoraproject.org/paste/~cSehNxN77eGxprxliE3dQ/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2697.0KiB)... https://da.gd/pRDI -> https://paste.fedoraproject.org/paste/-jJF5TRug~pcQ7syEK~qnA/ ---- test_logs/basic-suite-4.1/ post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/7mkNW -> https://paste.fedoraproject.org/paste/PjqtJIsVK~1UzHd1PtjUJA/ ---- On Mon, Aug 14, 2017 at 2:02 PM, Marc Young <3vilpenguin@gmail.com> wrote:
rpm -q python-ovirt-engine-sdk4
Mine is outdated:
$ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64
Ill update that and run again.
Logs (better to send too many than not enough sometimes). Ovirt engine logs are there.
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/tuned/tuned.log Uploading (3.5KiB)... https://da.gd/LiG0 -> https://paste.fedoraproject.org/paste/ wed7pYARTmHHTHcf0DvVAQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/audit/audit.log Uploading (401.2KiB)... https://da.gd/tgSw -> https://paste.fedoraproject.org/paste/ WEqegOFiLpiNGCv0hvY0ww/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/yum.log Uploading (24.3KiB)... https://da.gd/BzoMg -> https://paste.fedoraproject.org/paste/qla7z~ hRsym8kkCr6NlXFg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/setup-ds.log Uploading (37.4KiB)... https://da.gd/1HHKA -> https://paste.fedoraproject.org/paste/ 4nRRDoC8lrv5CTxbS6TCxQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine/setup/ovirt- engine-setup-20170814074645-vzih31.log Uploading (5544.4KiB)... https://da.gd/xYpYl -> https://paste.fedoraproject.org/paste/ goz7RQVvusLXjiiheLD5xg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.3KiB)... https://da.gd/4Y6pE -> https://paste.fedoraproject.org/paste/ OCqWe6tjGkrNOrTkqex3mg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (34.1KiB)... https://da.gd/PZm7d -> https://paste.fedoraproject.org/paste/ HpZOi8LTrgcWhKqfPeowPA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2724.0KiB)... https://da.gd/pIOO -> https://paste.fedoraproject.org/paste/ ufbtnfZOfSmun8sbOCs1xg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/Xf5R -> https://paste.fedoraproject. org/paste/MO-dlfNQXi19Tla3T~QXsA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/snmptrapd.log Uploading (0.1KiB)... https://da.gd/cF48 -> https://paste.fedoraproject. org/paste/NYqECCL1QVihcBQV~VWDsQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-host1/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/G5oC -> https://paste.fedoraproject.org/paste/ JgvhOLoA7cEO2Q3ESjP5Aw/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-host1/_var_log/audit/audit.log Uploading (183.1KiB)... https://da.gd/9bjYn -> https://paste.fedoraproject.org/paste/ Ado1joqnZxkPL1us6UeOwQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-host1/_var_log/yum.log Uploading (18.3KiB)... https://da.gd/i5ki -> https://paste.fedoraproject.org/paste/ 9sxQziGLRqw1tLHKfBAi6Q/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-host0/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/5Iiu4 -> https://paste.fedoraproject. org/paste/RaqsqKQdfgcTP1o2Epo~Pg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-host0/_var_log/audit/audit.log Uploading (179.9KiB)... https://da.gd/MyZ7 -> https://paste.fedoraproject.org/paste/ x7T7ZkIS3g5JE2dj028AVg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-host0/_var_log/yum.log Uploading (17.4KiB)... https://da.gd/Gmux -> https://paste.fedoraproject.org/paste/ kORmR4ShCv1EVGtTWSu2Zw/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs/lago.log Uploading (1198.1KiB)... https://da.gd/vzVCA -> https://paste.fedoraproject.org/paste/TuQiMEro23I~ OnFcM7wG-g/ ----
On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin@gmail.com> wrote:
You should have all logs needed @/home/myoung/repos/github/ovi
rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap .py/lago_logs
The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. Heres all logs for basic suite 4.1 that I know how to find:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
The engine and hosts logs are usually under the test_logs, which should be in the same subdir as lago_logs directory. Also, can you verify you have the latest python v4 SDK installled?
rpm -q python-ovirt-engine-sdk4
If you don't have it installed, you can get it directly from the build job ( which acts also as yum repo ):
el7 build - http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_4.1_bu ild-artifacts-el7-x86_64/lastSuccessfulBuild/artifact/exported-artifacts/
fc26 build is available only for master [1] but should also work.
There are also fc24/25 builds, just follow the same pattern to find them on Jenkins.
[1] http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_ma ster_build-artifacts-fc26-x86_64/lastSuccessfulBuild/artifac t/exported-artifacts/
$ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; done Uploading (1355.0KiB)... https://da.gd/OIlR -> https://paste.fedoraproject.or g/paste/9Xn2RuDgF2pY2I0CouBbGg/ Uploading (1154.0KiB)... https://da.gd/0Cti -> https://paste.fedoraproject.or g/paste/uJqvAiDSblnmN8OA6i5tXA/ Uploading (1377.8KiB)... https://da.gd/GpDIf -> https://paste.fedoraproject.or g/paste/~Cn6IJgbvateJfFulDNFCg/ Uploading (1176.7KiB)... https://da.gd/skhBy -> https://paste.fedoraproject.or g/paste/HIpEVm4TRPpH5nJCOyUfWA/ Uploading (1397.1KiB)... https://da.gd/fEtF -> https://paste.fedoraproject.or g/paste/cuycP~u9D~gJKNPsQ5NH2g/ Uploading (1198.1KiB)... https://da.gd/1zNm -> https://paste.fedoraproject.or g/paste/5~yEDwCSr~tdxHT~XUaLPg/
On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Full log is just as uninteresting: https://paste.f edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
I was actually hoping for the engine log. Y.
On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
> Thanks for clarifying, that makes sense in hindsight since its > testing things in their entirety. This is more of a rabbit hole for me as > Im really just trying to learn Lago by using the OST project (since I'll be > duplicating a lot of the setup). > > stdout logs from a run: https://pastebin.com/KBDaCCYp >
Can you send engine log so we'll understand why the host installation failed? You should have all logs needed @/home/myoung/repos/github/ovi rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap .py/lago_logs
Y.
> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote: > >> >> >> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> >> wrote: >> >>> Ill try to get some detailed log files later, but fwiw I'm not >>> running the hc suites (afaik, Im still getting inundated with the system >>> tests/lago). The link I used for Jenkins was just to try to pull the latest >>> 'passing' hash for ovirt-system-tests >>> >> >> I see, its a bit more complicated than that I'm afraid, the HASH >> you see is probably on for OST code itself, while the suites can fail on >> any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so >> its more than just a single hash, but rather a list of RPMs and their >> versions, together with hash of OST ( for the tests themselves ) and also >> OS updates. >> >> >>> >>> The errors have all been from master branch on the basic suite for >>> 4.1 via: >>> >>> $ ./run_suite.sh basic-suite-4.1/ >>> >>> >> OK, then we need to investigate it and understand why add host >> fails on 4.1, please provide logs when possible. >> >> BTW, you can also try running the manual job [1], but you'll need >> to provide it with custom yum repo URL with your built artifacts to test >> your code. >> There is a section on it on the OST readthedocs page. >> >> [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job >> /ovirt-system-tests_manual/ >> >> >>> >>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> >>> wrote: >>> >>>> I see you run the 'hc' suite, which means hyperconverged. This >>>> suite is running an hosted engine on gluster storage, It's a more complex >>>> suite than the basic one, and prone to more errors, however it should still >>>> work, but if you don't require it specifically, I would recommend running >>>> the basic suite, which should be easier to debug and also quicker to run. >>>> >>>> I'm also adding hc maintainer so she can check, can you share the >>>> link to the Jenkins job you're running? Or the log files if you're running >>>> locally. >>>> >>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> >>>> wrote: >>>> >>>>> Actually I spoke too soon, still fails: >>>>> >>>>> + lago ovirt runtest /home/myoung/repos/github/ovir >>>>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >>>>> @ Run test: 002_bootstrap.py: >>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>>>> '^setup\\.py$'] >>>>> # print_api_ver: >>>>> # print_api_ver: Success (in 0:00:00) >>>>> # add_dc: >>>>> # add_dc: Success (in 0:00:43) >>>>> # add_cluster: >>>>> # add_cluster: Success (in 0:00:03) >>>>> # add_hosts: >>>>> dd >>>>> * Collect artifacts: >>>>> * Collect artifacts: Success (in 0:01:14) >>>>> # add_hosts: Success (in 0:16:36) >>>>> # Results located at /home/myoung/repos/github/ovir >>>>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j >>>>> unit.xml >>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>>>> Error occured, aborting >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", >>>>> line 360, in do_run >>>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >>>>> line 184, in do_run >>>>> self._do_run(**vars(args)) >>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>> 501, in wrapper >>>>> return func(*args, **kwargs) >>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>> 512, in wrapper >>>>> return func(*args, prefix=prefix, **kwargs) >>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", >>>>> line 99, in do_ovirt_runtest >>>>> raise RuntimeError('Some tests failed') >>>>> RuntimeError: Some tests failed >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young < >>>>> 3vilpenguin@gmail.com> wrote: >>>>> >>>>>> Edit: reply-all >>>>>> >>>>>> It does, every time. >>>>>> I got it to pass by using the last passing revision from >>>>>> Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), >>>>>> noted from http://jenkins.ovirt.org/job/s >>>>>> ystem-tests_hc-basic-suite-master/15/consoleFull >>>>>> >>>>>> Not sure if it's a true race condition, that build in Jenkins >>>>>> has failed consistently since #15 >>>>>> >>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young < >>>>>>> 3vilpenguin@gmail.com> wrote: >>>>>>> >>>>>>>> What's the fix for this for those of us using >>>>>>>> ovirt-system-tests? >>>>>>>> >>>>>>>> Im trying to adapt some of the code for testing a third party >>>>>>>> tool, but master is still failing on 002_bootstrap >>>>>>>> >>>>>>> >>>>>>> It fails consistently? >>>>>>> AFAIK this is a race condition that happens maybe once a week, >>>>>>> can you share your logs? is it also failing on add_secondary_storage? >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>>>>>> bzlotnik@redhat.com> wrote: >>>>>>>> >>>>>>>>> I'm not sure it's related since the LSM test runs much >>>>>>>>> later, in 004 >>>>>>>>> >>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Adding Allon & Benny. >>>>>>>>>> >>>>>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>>>>> we've been handling in [1]? >>>>>>>>>> If this is the case, we agreed to disable the test next >>>>>>>>>> time it fails, as the current workarounds with sleep isn't enough. >>>>>>>>>> >>>>>>>>>> Can you confirm this is the case, and so we'll have to >>>>>>>>>> disable this test until one of the RFEs described in [1] is merged? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>>>> >>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>>> >>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>>>> >> >>>>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>>>>>> ] >>>>>>>>>>> >> >>>>>>>>>>> >> Link to suspected patches: >>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > This patch adds missing log when resizing an online disk >>>>>>>>>>> - why do you think >>>>>>>>>>> > it is related to the failure? >>>>>>>>>>> >>>>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>>>> (The test was equivalent to running the manual job with >>>>>>>>>>> just this patch) >>>>>>>>>>> >>>>>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>>>>> conditions. >>>>>>>>>>> Since the code in the patch seemed to be related to >>>>>>>>>>> storage I didn't >>>>>>>>>>> want to just assume that. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Barak Korren >>>>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>>>> Red Hat EMEA >>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list >>>>>>>>>>> Devel@ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Eyal edri >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ASSOCIATE MANAGER >>>>>>>>>> >>>>>>>>>> RHV DevOps >>>>>>>>>> >>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>>> <https://redhat.com/trusted> >>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list >>>>>>>>> Devel@ovirt.org >>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Eyal edri >>>>>>> >>>>>>> >>>>>>> ASSOCIATE MANAGER >>>>>>> >>>>>>> RHV DevOps >>>>>>> >>>>>>> EMEA VIRTUALIZATION R&D >>>>>>> >>>>>>> >>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>> <https://redhat.com/trusted> >>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>> >>>>>> >>>>>> >>>>> >>> >> >> >> -- >> >> Eyal edri >> >> >> ASSOCIATE MANAGER >> >> RHV DevOps >> >> EMEA VIRTUALIZATION R&D >> >> >> Red Hat EMEA <https://www.redhat.com/> >> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >> <https://redhat.com/trusted> >> phone: +972-9-7692018 <+972%209-769-2018> >> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > > > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
Hi Marc, Some of the logs failed to extract, so I'm not sure what went wrong. Specifically, host-deploy directory and vdms logs from the hosts. (They should all be under test_logs, I would just tar.gz the entire directory and send it next time). As "add hosts" step failed on timeout(15 minutes) - I suspect is due to a missing package or network issue(some packages might be downloaded from the VDMS hosts during that stage). So: 1. Could you retry? is this consistent? 2. If so, can you please upload the entire log directory? If we discover this indeed is a network issue, you could just increase the timeout. Thanks, Nadav. On Tue, Aug 15, 2017 at 5:28 AM, Marc Young <3vilpenguin@gmail.com> wrote:
After updating the python sdk:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.6-2.20170712git1b99f36.el7.centos.x86_64
I get more but different errors[1].
Then the lago log[2] All engine Logs are even further[3]
[1]
# add_cluster: Success (in 0:00:03) # add_hosts: * Collect artifacts: - [Thread-5] lago-basic-suite-4-1-host1: ERROR (in 0:00:19) - [Thread-4] lago-basic-suite-4-1-engine: ERROR (in 0:00:20) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout
Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout * Collect artifacts: ERROR (in 0:00:33)
# add_hosts: ERROR (in 0:15:52) # Collect artifacts: * [Thread-19] lago-basic-suite-4-1-engine: * [Thread-20] lago-basic-suite-4-1-host1: * [Thread-21] lago-basic-suite-4-1-host0: * [Thread-21] lago-basic-suite-4-1-host0: Success (in 0:00:03) * [Thread-20] lago-basic-suite-4-1-host1: Success (in 0:00:05) * [Thread-19] lago-basic-suite-4-1-engine: Success (in 0:00:16) # Collect artifacts: Success (in 0:00:16) # Results located at /home/myoung/repos/github/ovirt-system-tests/deployment-basic-suite-4.1/default/002_bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: ERROR (in 0:16:57) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed + res=1 + cd - /home/myoung/repos/github/ovirt-system-tests + return 1 + failed=true + env_collect /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py + local tests_out_dir=/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py + echo '#########################' ######################### + [[ -e /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1 ]] + cd /home/myoung/repos/github/ovirt-system-tests/deployment-basic-suite-4.1/current + lago collect --output /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py @ Collect artifacts: # [Thread-1] lago-basic-suite-4-1-engine: # [Thread-2] lago-basic-suite-4-1-host1: # [Thread-3] lago-basic-suite-4-1-host0: # [Thread-3] lago-basic-suite-4-1-host0: Success (in 0:00:02) # [Thread-2] lago-basic-suite-4-1-host1: Success (in 0:00:03) # [Thread-1] lago-basic-suite-4-1-engine: Success (in 0:00:06) @ Collect artifacts: Success (in 0:00:07) + cp -a logs /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs + cd - /home/myoung/repos/github/ovirt-system-tests + true + echo '@@@@ ERROR: Failed running /home/myoung/repos/github/ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py' @@@@ ERROR: Failed running /home/myoung/repos/github/ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py + return 1
[2]
$ cat './test_logs/basic-suite-4.1/post-002_bootstrap.py.2017-08-14_21:55:00/lago_logs/lago.log' | fpaste Uploading (1202.8KiB)... https://da.gd/6aw2 -> https://paste.fedoraproject.org/paste/956xJBRY-m1D9T7giQktHQ/
[3]
$ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log | grep ovirt-engine); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20170814212716-ql8ekq.log Uploading (5544.7KiB)... https://da.gd/QLkSD -> https://paste.fedoraproject.org/paste/tQEEAfL3TQk7nYhrdyQIVQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.5KiB)... https://da.gd/eX4c6 -> https://paste.fedoraproject.org/paste/7YFWv6uVzARJ8q0iCAhUNQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (33.9KiB)... https://da.gd/n1JTd -> https://paste.fedoraproject.org/paste/~cSehNxN77eGxprxliE3dQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2697.0KiB)... https://da.gd/pRDI -> https://paste.fedoraproject.org/paste/-jJF5TRug~pcQ7syEK~qnA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/7mkNW -> https://paste.fedoraproject.org/paste/PjqtJIsVK~1UzHd1PtjUJA/ ----
On Mon, Aug 14, 2017 at 2:02 PM, Marc Young <3vilpenguin@gmail.com> wrote:
rpm -q python-ovirt-engine-sdk4
Mine is outdated:
$ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64
Ill update that and run again.
Logs (better to send too many than not enough sometimes). Ovirt engine logs are there.
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log); do echo $i; cat $i | fpaste; echo ----; done
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/tuned/tuned.log Uploading (3.5KiB)... https://da.gd/LiG0 -> https://paste.fedoraproject.org/paste/wed7pYARTmHHTHcf0DvVAQ/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/audit/audit.log Uploading (401.2KiB)... https://da.gd/tgSw -> https://paste.fedoraproject.org/paste/WEqegOFiLpiNGCv0hvY0ww/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/yum.log Uploading (24.3KiB)... https://da.gd/BzoMg -> https://paste.fedoraproject.org/paste/qla7z~hRsym8kkCr6NlXFg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/setup-ds.log Uploading (37.4KiB)... https://da.gd/1HHKA -> https://paste.fedoraproject.org/paste/4nRRDoC8lrv5CTxbS6TCxQ/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20170814074645-vzih31.log Uploading (5544.4KiB)... https://da.gd/xYpYl -> https://paste.fedoraproject.org/paste/goz7RQVvusLXjiiheLD5xg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.3KiB)... https://da.gd/4Y6pE -> https://paste.fedoraproject.org/paste/OCqWe6tjGkrNOrTkqex3mg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (34.1KiB)... https://da.gd/PZm7d -> https://paste.fedoraproject.org/paste/HpZOi8LTrgcWhKqfPeowPA/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2724.0KiB)... https://da.gd/pIOO -> https://paste.fedoraproject.org/paste/ufbtnfZOfSmun8sbOCs1xg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/Xf5R -> https://paste.fedoraproject.org/paste/MO-dlfNQXi19Tla3T~QXsA/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/snmptrapd.log Uploading (0.1KiB)... https://da.gd/cF48 -> https://paste.fedoraproject.org/paste/NYqECCL1QVihcBQV~VWDsQ/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/G5oC -> https://paste.fedoraproject.org/paste/JgvhOLoA7cEO2Q3ESjP5Aw/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/audit/audit.log Uploading (183.1KiB)... https://da.gd/9bjYn -> https://paste.fedoraproject.org/paste/Ado1joqnZxkPL1us6UeOwQ/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/yum.log Uploading (18.3KiB)... https://da.gd/i5ki -> https://paste.fedoraproject.org/paste/9sxQziGLRqw1tLHKfBAi6Q/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/5Iiu4 -> https://paste.fedoraproject.org/paste/RaqsqKQdfgcTP1o2Epo~Pg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/audit/audit.log Uploading (179.9KiB)... https://da.gd/MyZ7 -> https://paste.fedoraproject.org/paste/x7T7ZkIS3g5JE2dj028AVg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/yum.log Uploading (17.4KiB)... https://da.gd/Gmux -> https://paste.fedoraproject.org/paste/kORmR4ShCv1EVGtTWSu2Zw/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs/lago.log Uploading (1198.1KiB)... https://da.gd/vzVCA -> https://paste.fedoraproject.org/paste/TuQiMEro23I~OnFcM7wG-g/ ----
On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin@gmail.com> wrote:
You should have all logs needed @/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs
The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. Heres all logs for basic suite 4.1 that I know how to find:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
The engine and hosts logs are usually under the test_logs, which should be in the same subdir as lago_logs directory. Also, can you verify you have the latest python v4 SDK installled?
rpm -q python-ovirt-engine-sdk4
If you don't have it installed, you can get it directly from the build job ( which acts also as yum repo ):
el7 build - http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_4.1_build-artifacts-el...
fc26 build is available only for master [1] but should also work.
There are also fc24/25 builds, just follow the same pattern to find them on Jenkins.
[1] http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_master_build-artifacts...
$ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; done Uploading (1355.0KiB)... https://da.gd/OIlR -> https://paste.fedoraproject.org/paste/9Xn2RuDgF2pY2I0CouBbGg/ Uploading (1154.0KiB)... https://da.gd/0Cti -> https://paste.fedoraproject.org/paste/uJqvAiDSblnmN8OA6i5tXA/ Uploading (1377.8KiB)... https://da.gd/GpDIf -> https://paste.fedoraproject.org/paste/~Cn6IJgbvateJfFulDNFCg/ Uploading (1176.7KiB)... https://da.gd/skhBy -> https://paste.fedoraproject.org/paste/HIpEVm4TRPpH5nJCOyUfWA/ Uploading (1397.1KiB)... https://da.gd/fEtF -> https://paste.fedoraproject.org/paste/cuycP~u9D~gJKNPsQ5NH2g/ Uploading (1198.1KiB)... https://da.gd/1zNm -> https://paste.fedoraproject.org/paste/5~yEDwCSr~tdxHT~XUaLPg/
On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Full log is just as uninteresting: https://paste.fedoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
I was actually hoping for the engine log. Y.
On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote: > > > > On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> > wrote: >> >> Thanks for clarifying, that makes sense in hindsight since its >> testing things in their entirety. This is more of a rabbit hole for me as Im >> really just trying to learn Lago by using the OST project (since I'll be >> duplicating a lot of the setup). >> >> stdout logs from a run: https://pastebin.com/KBDaCCYp > > > Can you send engine log so we'll understand why the host installation > failed? > You should have all logs needed > @/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs > > Y. > >> >> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote: >>> >>> >>> >>> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> >>> wrote: >>>> >>>> Ill try to get some detailed log files later, but fwiw I'm not >>>> running the hc suites (afaik, Im still getting inundated with the system >>>> tests/lago). The link I used for Jenkins was just to try to pull the latest >>>> 'passing' hash for ovirt-system-tests >>> >>> >>> I see, its a bit more complicated than that I'm afraid, the HASH >>> you see is probably on for OST code itself, while the suites can fail on any >>> oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than >>> just a single hash, but rather a list of RPMs and their versions, together >>> with hash of OST ( for the tests themselves ) and also OS updates. >>> >>>> >>>> >>>> The errors have all been from master branch on the basic suite for >>>> 4.1 via: >>>> >>>> $ ./run_suite.sh basic-suite-4.1/ >>> >>> >>> OK, then we need to investigate it and understand why add host >>> fails on 4.1, please provide logs when possible. >>> >>> BTW, you can also try running the manual job [1], but you'll need >>> to provide it with custom yum repo URL with your built artifacts to test >>> your code. >>> There is a section on it on the OST readthedocs page. >>> >>> [1] >>> http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_... >>> >>>> >>>> >>>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> >>>> wrote: >>>>> >>>>> I see you run the 'hc' suite, which means hyperconverged. This >>>>> suite is running an hosted engine on gluster storage, It's a more complex >>>>> suite than the basic one, and prone to more errors, however it should still >>>>> work, but if you don't require it specifically, I would recommend running >>>>> the basic suite, which should be easier to debug and also quicker to run. >>>>> >>>>> I'm also adding hc maintainer so she can check, can you share the >>>>> link to the Jenkins job you're running? Or the log files if you're running >>>>> locally. >>>>> >>>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> >>>>> wrote: >>>>>> >>>>>> Actually I spoke too soon, still fails: >>>>>> >>>>>> + lago ovirt runtest >>>>>> /home/myoung/repos/github/ovirt-system-tests/vagrant/test-scenarios/002_bootstrap.py >>>>>> @ Run test: 002_bootstrap.py: >>>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>>>>> '^setup\\.py$'] >>>>>> # print_api_ver: >>>>>> # print_api_ver: Success (in 0:00:00) >>>>>> # add_dc: >>>>>> # add_dc: Success (in 0:00:43) >>>>>> # add_cluster: >>>>>> # add_cluster: Success (in 0:00:03) >>>>>> # add_hosts: >>>>>> dd >>>>>> * Collect artifacts: >>>>>> * Collect artifacts: Success (in 0:01:14) >>>>>> # add_hosts: Success (in 0:16:36) >>>>>> # Results located at >>>>>> /home/myoung/repos/github/ovirt-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml >>>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>>>>> Error occured, aborting >>>>>> Traceback (most recent call last): >>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>>>>> 360, in do_run >>>>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >>>>>> line 184, in do_run >>>>>> self._do_run(**vars(args)) >>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>> 501, in wrapper >>>>>> return func(*args, **kwargs) >>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>> 512, in wrapper >>>>>> return func(*args, prefix=prefix, **kwargs) >>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line >>>>>> 99, in do_ovirt_runtest >>>>>> raise RuntimeError('Some tests failed') >>>>>> RuntimeError: Some tests failed >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young >>>>>> <3vilpenguin@gmail.com> wrote: >>>>>>> >>>>>>> Edit: reply-all >>>>>>> >>>>>>> It does, every time. >>>>>>> I got it to pass by using the last passing revision from >>>>>>> Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from >>>>>>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-master/15/consoleFu... >>>>>>> >>>>>>> Not sure if it's a true race condition, that build in Jenkins >>>>>>> has failed consistently since #15 >>>>>>> >>>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> >>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young >>>>>>>> <3vilpenguin@gmail.com> wrote: >>>>>>>>> >>>>>>>>> What's the fix for this for those of us using >>>>>>>>> ovirt-system-tests? >>>>>>>>> >>>>>>>>> Im trying to adapt some of the code for testing a third party >>>>>>>>> tool, but master is still failing on 002_bootstrap >>>>>>>> >>>>>>>> >>>>>>>> It fails consistently? >>>>>>>> AFAIK this is a race condition that happens maybe once a week, >>>>>>>> can you share your logs? is it also failing on add_secondary_storage? >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik >>>>>>>>> <bzlotnik@redhat.com> wrote: >>>>>>>>>> >>>>>>>>>> I'm not sure it's related since the LSM test runs much >>>>>>>>>> later, in 004 >>>>>>>>>> >>>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Adding Allon & Benny. >>>>>>>>>>> >>>>>>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>>>>>> we've been handling in [1]? >>>>>>>>>>> If this is the case, we agreed to disable the test next >>>>>>>>>>> time it fails, as the current workarounds with sleep isn't enough. >>>>>>>>>>> >>>>>>>>>>> Can you confirm this is the case, and so we'll have to >>>>>>>>>>> disable this test until one of the RFEs described in [1] is merged? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren >>>>>>>>>>> <bkorren@redhat.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren >>>>>>>>>>>> > <bkorren@redhat.com> wrote: >>>>>>>>>>>> >> >>>>>>>>>>>> >> Test failed: [ >>>>>>>>>>>> >> 002_bootstrap.add_secondary_storage_domains ] >>>>>>>>>>>> >> >>>>>>>>>>>> >> Link to suspected patches: >>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > This patch adds missing log when resizing an online disk >>>>>>>>>>>> > - why do you think >>>>>>>>>>>> > it is related to the failure? >>>>>>>>>>>> >>>>>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>>>>> (The test was equivalent to running the manual job with >>>>>>>>>>>> just this patch) >>>>>>>>>>>> >>>>>>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>>>>>> conditions. >>>>>>>>>>>> Since the code in the patch seemed to be related to >>>>>>>>>>>> storage I didn't >>>>>>>>>>>> want to just assume that. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Barak Korren >>>>>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>>>>> Red Hat EMEA >>>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Devel mailing list >>>>>>>>>>>> Devel@ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Eyal edri >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ASSOCIATE MANAGER >>>>>>>>>>> >>>>>>>>>>> RHV DevOps >>>>>>>>>>> >>>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Red Hat EMEA >>>>>>>>>>> >>>>>>>>>>> TRIED. TESTED. TRUSTED. >>>>>>>>>>> phone: +972-9-7692018 >>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list >>>>>>>>>> Devel@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Eyal edri >>>>>>>> >>>>>>>> >>>>>>>> ASSOCIATE MANAGER >>>>>>>> >>>>>>>> RHV DevOps >>>>>>>> >>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>> >>>>>>>> >>>>>>>> Red Hat EMEA >>>>>>>> >>>>>>>> TRIED. TESTED. TRUSTED. >>>>>>>> phone: +972-9-7692018 >>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>> >>>>>>> >>>>>> >>>> >>> >>> >>> >>> -- >>> >>> Eyal edri >>> >>> >>> ASSOCIATE MANAGER >>> >>> RHV DevOps >>> >>> EMEA VIRTUALIZATION R&D >>> >>> >>> Red Hat EMEA >>> >>> TRIED. TESTED. TRUSTED. >>> phone: +972-9-7692018 >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >> >> >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel > >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA
TRIED. TESTED. TRUSTED. phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
On Tue, Aug 15, 2017 at 5:28 AM, Marc Young <3vilpenguin@gmail.com> wrote:
After updating the python sdk:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.6-2.20170712git1b99f36.el7.centos.x86_64
I get more but different errors[1].
Then the lago log[2] All engine Logs are even further[3]
OK, understood the issue. Your host installation is for some reason very very slow - the test just times out after 15 minutes. It's why the Engine log is a bit cut. It begins here: 2017-08-14 *21:38:32*,143-04 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (default task-10) [10728bdc-0cdc-45ba-90f8-7a78b0b79d6e] Connecting 'root@lago-basic-suite-4-1-host0' And it still installing here: 2017-08-14 *21:55:01*,438-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [76cb1954] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Correlation ID: 76cb1954, Call Stack: null, Custom Event ID: -1, Message: Installing Host lago-basic-suite-4-1-host0. Yum install: 239/364: python-netaddr-0.7.5-7.el7.noarch. But ovirt-system-tests already given up. On my laptop it takes ~3 minutes - perhaps you are running on qemu and not kvm, or something similar? It shouldn't be that slow. Y.
[1]
# add_cluster: Success (in 0:00:03) # add_hosts: * Collect artifacts: - [Thread-5] lago-basic-suite-4-1-host1: ERROR (in 0:00:19) - [Thread-4] lago-basic-suite-4-1-engine: ERROR (in 0:00:20) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout
Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout * Collect artifacts: ERROR (in 0:00:33)
# add_hosts: ERROR (in 0:15:52) # Collect artifacts: * [Thread-19] lago-basic-suite-4-1-engine: * [Thread-20] lago-basic-suite-4-1-host1: * [Thread-21] lago-basic-suite-4-1-host0: * [Thread-21] lago-basic-suite-4-1-host0: Success (in 0:00:03) * [Thread-20] lago-basic-suite-4-1-host1: Success (in 0:00:05) * [Thread-19] lago-basic-suite-4-1-engine: Success (in 0:00:16) # Collect artifacts: Success (in 0:00:16) # Results located at /home/myoung/repos/github/ ovirt-system-tests/deployment-basic-suite-4.1/default/002_ bootstrap.py.junit.xml @ Run test: 002_bootstrap.py: ERROR (in 0:16:57) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed + res=1 + cd - /home/myoung/repos/github/ovirt-system-tests + return 1 + failed=true + env_collect /home/myoung/repos/github/ovirt-system-tests/test_logs/ basic-suite-4.1/post-002_bootstrap.py + local tests_out_dir=/home/myoung/repos/github/ovirt-system- tests/test_logs/basic-suite-4.1/post-002_bootstrap.py + echo '#########################' ######################### + [[ -e /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1 ]] + cd /home/myoung/repos/github/ovirt-system-tests/deployment- basic-suite-4.1/current + lago collect --output /home/myoung/repos/github/ ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py @ Collect artifacts: # [Thread-1] lago-basic-suite-4-1-engine: # [Thread-2] lago-basic-suite-4-1-host1: # [Thread-3] lago-basic-suite-4-1-host0: # [Thread-3] lago-basic-suite-4-1-host0: Success (in 0:00:02) # [Thread-2] lago-basic-suite-4-1-host1: Success (in 0:00:03) # [Thread-1] lago-basic-suite-4-1-engine: Success (in 0:00:06) @ Collect artifacts: Success (in 0:00:07) + cp -a logs /home/myoung/repos/github/ovirt-system-tests/test_logs/ basic-suite-4.1/post-002_bootstrap.py/lago_logs + cd - /home/myoung/repos/github/ovirt-system-tests + true + echo '@@@@ ERROR: Failed running /home/myoung/repos/github/ ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py' @@@@ ERROR: Failed running /home/myoung/repos/github/ ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py + return 1
[2]
$ cat './test_logs/basic-suite-4.1/post-002_bootstrap.py.2017-08- 14_21:55:00/lago_logs/lago.log' | fpaste Uploading (1202.8KiB)... https://da.gd/6aw2 -> https://paste.fedoraproject.org/paste/956xJBRY- m1D9T7giQktHQ/
[3]
$ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log | grep ovirt-engine); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine/setup/ovirt- engine-setup-20170814212716-ql8ekq.log Uploading (5544.7KiB)... https://da.gd/QLkSD -> https://paste.fedoraproject.org/paste/ tQEEAfL3TQk7nYhrdyQIVQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.5KiB)... https://da.gd/eX4c6 -> https://paste.fedoraproject.org/paste/ 7YFWv6uVzARJ8q0iCAhUNQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (33.9KiB)... https://da.gd/n1JTd -> https://paste.fedoraproject.org/paste/~ cSehNxN77eGxprxliE3dQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2697.0KiB)... https://da.gd/pRDI -> https://paste.fedoraproject. org/paste/-jJF5TRug~pcQ7syEK~qnA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic- suite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/7mkNW -> https://paste.fedoraproject.org/paste/PjqtJIsVK~ 1UzHd1PtjUJA/ ----
On Mon, Aug 14, 2017 at 2:02 PM, Marc Young <3vilpenguin@gmail.com> wrote:
rpm -q python-ovirt-engine-sdk4
Mine is outdated:
$ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64
Ill update that and run again.
Logs (better to send too many than not enough sometimes). Ovirt engine logs are there.
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/tuned/tuned.log Uploading (3.5KiB)... https://da.gd/LiG0 -> https://paste.fedoraproject.or g/paste/wed7pYARTmHHTHcf0DvVAQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/audit/audit.log Uploading (401.2KiB)... https://da.gd/tgSw -> https://paste.fedoraproject.or g/paste/WEqegOFiLpiNGCv0hvY0ww/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/yum.log Uploading (24.3KiB)... https://da.gd/BzoMg -> https://paste.fedoraproject.or g/paste/qla7z~hRsym8kkCr6NlXFg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/setup-ds.log Uploading (37.4KiB)... https://da.gd/1HHKA -> https://paste.fedoraproject.or g/paste/4nRRDoC8lrv5CTxbS6TCxQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine- setup-20170814074645-vzih31.log Uploading (5544.4KiB)... https://da.gd/xYpYl -> https://paste.fedoraproject.or g/paste/goz7RQVvusLXjiiheLD5xg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.3KiB)... https://da.gd/4Y6pE -> https://paste.fedoraproject.or g/paste/OCqWe6tjGkrNOrTkqex3mg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (34.1KiB)... https://da.gd/PZm7d -> https://paste.fedoraproject.or g/paste/HpZOi8LTrgcWhKqfPeowPA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2724.0KiB)... https://da.gd/pIOO -> https://paste.fedoraproject.or g/paste/ufbtnfZOfSmun8sbOCs1xg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/Xf5R -> https://paste.fedoraproject.or g/paste/MO-dlfNQXi19Tla3T~QXsA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/snmptrapd.log Uploading (0.1KiB)... https://da.gd/cF48 -> https://paste.fedoraproject.or g/paste/NYqECCL1QVihcBQV~VWDsQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/G5oC -> https://paste.fedoraproject.or g/paste/JgvhOLoA7cEO2Q3ESjP5Aw/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/audit/audit.log Uploading (183.1KiB)... https://da.gd/9bjYn -> https://paste.fedoraproject.or g/paste/Ado1joqnZxkPL1us6UeOwQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/yum.log Uploading (18.3KiB)... https://da.gd/i5ki -> https://paste.fedoraproject.or g/paste/9sxQziGLRqw1tLHKfBAi6Q/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/5Iiu4 -> https://paste.fedoraproject.or g/paste/RaqsqKQdfgcTP1o2Epo~Pg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/audit/audit.log Uploading (179.9KiB)... https://da.gd/MyZ7 -> https://paste.fedoraproject.or g/paste/x7T7ZkIS3g5JE2dj028AVg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/yum.log Uploading (17.4KiB)... https://da.gd/Gmux -> https://paste.fedoraproject.or g/paste/kORmR4ShCv1EVGtTWSu2Zw/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs/lago.log Uploading (1198.1KiB)... https://da.gd/vzVCA -> https://paste.fedoraproject.or g/paste/TuQiMEro23I~OnFcM7wG-g/ ----
On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin@gmail.com> wrote:
You should have all logs needed @/home/myoung/repos/github/ovi
rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap .py/lago_logs
The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. Heres all logs for basic suite 4.1 that I know how to find:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
The engine and hosts logs are usually under the test_logs, which should be in the same subdir as lago_logs directory. Also, can you verify you have the latest python v4 SDK installled?
rpm -q python-ovirt-engine-sdk4
If you don't have it installed, you can get it directly from the build job ( which acts also as yum repo ):
el7 build - http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_4.1_bu ild-artifacts-el7-x86_64/lastSuccessfulBuild/artifact/export ed-artifacts/
fc26 build is available only for master [1] but should also work.
There are also fc24/25 builds, just follow the same pattern to find them on Jenkins.
[1] http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_ma ster_build-artifacts-fc26-x86_64/lastSuccessfulBuild/artifac t/exported-artifacts/
$ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; done Uploading (1355.0KiB)... https://da.gd/OIlR -> https://paste.fedoraproject.or g/paste/9Xn2RuDgF2pY2I0CouBbGg/ Uploading (1154.0KiB)... https://da.gd/0Cti -> https://paste.fedoraproject.or g/paste/uJqvAiDSblnmN8OA6i5tXA/ Uploading (1377.8KiB)... https://da.gd/GpDIf -> https://paste.fedoraproject.or g/paste/~Cn6IJgbvateJfFulDNFCg/ Uploading (1176.7KiB)... https://da.gd/skhBy -> https://paste.fedoraproject.or g/paste/HIpEVm4TRPpH5nJCOyUfWA/ Uploading (1397.1KiB)... https://da.gd/fEtF -> https://paste.fedoraproject.or g/paste/cuycP~u9D~gJKNPsQ5NH2g/ Uploading (1198.1KiB)... https://da.gd/1zNm -> https://paste.fedoraproject.or g/paste/5~yEDwCSr~tdxHT~XUaLPg/
On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Full log is just as uninteresting: https://paste.f edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
I was actually hoping for the engine log. Y.
On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
> > > On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> > wrote: > >> Thanks for clarifying, that makes sense in hindsight since its >> testing things in their entirety. This is more of a rabbit hole for me as >> Im really just trying to learn Lago by using the OST project (since I'll be >> duplicating a lot of the setup). >> >> stdout logs from a run: https://pastebin.com/KBDaCCYp >> > > Can you send engine log so we'll understand why the host > installation failed? > You should have all logs needed @/home/myoung/repos/github/ovi > rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap > .py/lago_logs > > Y. > > >> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> >> wrote: >> >>> >>> >>> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com >>> > wrote: >>> >>>> Ill try to get some detailed log files later, but fwiw I'm not >>>> running the hc suites (afaik, Im still getting inundated with the system >>>> tests/lago). The link I used for Jenkins was just to try to pull the latest >>>> 'passing' hash for ovirt-system-tests >>>> >>> >>> I see, its a bit more complicated than that I'm afraid, the HASH >>> you see is probably on for OST code itself, while the suites can fail on >>> any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so >>> its more than just a single hash, but rather a list of RPMs and their >>> versions, together with hash of OST ( for the tests themselves ) and also >>> OS updates. >>> >>> >>>> >>>> The errors have all been from master branch on the basic suite >>>> for 4.1 via: >>>> >>>> $ ./run_suite.sh basic-suite-4.1/ >>>> >>>> >>> OK, then we need to investigate it and understand why add host >>> fails on 4.1, please provide logs when possible. >>> >>> BTW, you can also try running the manual job [1], but you'll need >>> to provide it with custom yum repo URL with your built artifacts to test >>> your code. >>> There is a section on it on the OST readthedocs page. >>> >>> [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job >>> /ovirt-system-tests_manual/ >>> >>> >>>> >>>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> >>>> wrote: >>>> >>>>> I see you run the 'hc' suite, which means hyperconverged. This >>>>> suite is running an hosted engine on gluster storage, It's a more complex >>>>> suite than the basic one, and prone to more errors, however it should still >>>>> work, but if you don't require it specifically, I would recommend running >>>>> the basic suite, which should be easier to debug and also quicker to run. >>>>> >>>>> I'm also adding hc maintainer so she can check, can you share >>>>> the link to the Jenkins job you're running? Or the log files if you're >>>>> running locally. >>>>> >>>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> >>>>> wrote: >>>>> >>>>>> Actually I spoke too soon, still fails: >>>>>> >>>>>> + lago ovirt runtest /home/myoung/repos/github/ovir >>>>>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >>>>>> @ Run test: 002_bootstrap.py: >>>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>>>>> '^setup\\.py$'] >>>>>> # print_api_ver: >>>>>> # print_api_ver: Success (in 0:00:00) >>>>>> # add_dc: >>>>>> # add_dc: Success (in 0:00:43) >>>>>> # add_cluster: >>>>>> # add_cluster: Success (in 0:00:03) >>>>>> # add_hosts: >>>>>> dd >>>>>> * Collect artifacts: >>>>>> * Collect artifacts: Success (in 0:01:14) >>>>>> # add_hosts: Success (in 0:16:36) >>>>>> # Results located at /home/myoung/repos/github/ovir >>>>>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j >>>>>> unit.xml >>>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>>>>> Error occured, aborting >>>>>> Traceback (most recent call last): >>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", >>>>>> line 360, in do_run >>>>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >>>>>> line 184, in do_run >>>>>> self._do_run(**vars(args)) >>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>> 501, in wrapper >>>>>> return func(*args, **kwargs) >>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>> 512, in wrapper >>>>>> return func(*args, prefix=prefix, **kwargs) >>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", >>>>>> line 99, in do_ovirt_runtest >>>>>> raise RuntimeError('Some tests failed') >>>>>> RuntimeError: Some tests failed >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young < >>>>>> 3vilpenguin@gmail.com> wrote: >>>>>> >>>>>>> Edit: reply-all >>>>>>> >>>>>>> It does, every time. >>>>>>> I got it to pass by using the last passing revision from >>>>>>> Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), >>>>>>> noted from http://jenkins.ovirt.org/job/s >>>>>>> ystem-tests_hc-basic-suite-master/15/consoleFull >>>>>>> >>>>>>> Not sure if it's a true race condition, that build in Jenkins >>>>>>> has failed consistently since #15 >>>>>>> >>>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young < >>>>>>>> 3vilpenguin@gmail.com> wrote: >>>>>>>> >>>>>>>>> What's the fix for this for those of us using >>>>>>>>> ovirt-system-tests? >>>>>>>>> >>>>>>>>> Im trying to adapt some of the code for testing a third >>>>>>>>> party tool, but master is still failing on 002_bootstrap >>>>>>>>> >>>>>>>> >>>>>>>> It fails consistently? >>>>>>>> AFAIK this is a race condition that happens maybe once a >>>>>>>> week, can you share your logs? is it also failing on add_secondary_storage? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>>>>>>> bzlotnik@redhat.com> wrote: >>>>>>>>> >>>>>>>>>> I'm not sure it's related since the LSM test runs much >>>>>>>>>> later, in 004 >>>>>>>>>> >>>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Adding Allon & Benny. >>>>>>>>>>> >>>>>>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>>>>>> we've been handling in [1]? >>>>>>>>>>> If this is the case, we agreed to disable the test next >>>>>>>>>>> time it fails, as the current workarounds with sleep isn't enough. >>>>>>>>>>> >>>>>>>>>>> Can you confirm this is the case, and so we'll have to >>>>>>>>>>> disable this test until one of the RFEs described in [1] is merged? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>>>>> >> >>>>>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>>>>>>> ] >>>>>>>>>>>> >> >>>>>>>>>>>> >> Link to suspected patches: >>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > This patch adds missing log when resizing an online >>>>>>>>>>>> disk - why do you think >>>>>>>>>>>> > it is related to the failure? >>>>>>>>>>>> >>>>>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>>>>> (The test was equivalent to running the manual job with >>>>>>>>>>>> just this patch) >>>>>>>>>>>> >>>>>>>>>>>> Then again this may also one of the usual SD testing race >>>>>>>>>>>> conditions. >>>>>>>>>>>> Since the code in the patch seemed to be related to >>>>>>>>>>>> storage I didn't >>>>>>>>>>>> want to just assume that. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Barak Korren >>>>>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>>>>> Red Hat EMEA >>>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Devel mailing list >>>>>>>>>>>> Devel@ovirt.org >>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Eyal edri >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ASSOCIATE MANAGER >>>>>>>>>>> >>>>>>>>>>> RHV DevOps >>>>>>>>>>> >>>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>>>> <https://redhat.com/trusted> >>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list >>>>>>>>>> Devel@ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Eyal edri >>>>>>>> >>>>>>>> >>>>>>>> ASSOCIATE MANAGER >>>>>>>> >>>>>>>> RHV DevOps >>>>>>>> >>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>> >>>>>>>> >>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>> <https://redhat.com/trusted> >>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>> >>> >>> -- >>> >>> Eyal edri >>> >>> >>> ASSOCIATE MANAGER >>> >>> RHV DevOps >>> >>> EMEA VIRTUALIZATION R&D >>> >>> >>> Red Hat EMEA <https://www.redhat.com/> >>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>> <https://redhat.com/trusted> >>> phone: +972-9-7692018 <+972%209-769-2018> >>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>> >> >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel >> > >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
Its in KVM. Ill bump the specs. Thanks for bearing with me, I'll see what I can do. It wasnt immediately obvious On Tue, Aug 15, 2017 at 4:24 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Tue, Aug 15, 2017 at 5:28 AM, Marc Young <3vilpenguin@gmail.com> wrote:
After updating the python sdk:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.6-2.20170712git1b99f36.el7.centos.x86_64
I get more but different errors[1].
Then the lago log[2] All engine Logs are even further[3]
OK, understood the issue. Your host installation is for some reason very very slow - the test just times out after 15 minutes. It's why the Engine log is a bit cut. It begins here: 2017-08-14 *21:38:32*,143-04 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (default task-10) [10728bdc-0cdc-45ba-90f8-7a78b0b79d6e] Connecting 'root@lago-basic-suite-4-1-host0'
And it still installing here: 2017-08-14 *21:55:01*,438-04 INFO [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [76cb1954] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Correlation ID: 76cb1954, Call Stack: null, Custom Event ID: -1, Message: Installing Host lago-basic-suite-4-1-host0. Yum install: 239/364: python-netaddr-0.7.5-7.el7.noarch.
But ovirt-system-tests already given up. On my laptop it takes ~3 minutes - perhaps you are running on qemu and not kvm, or something similar? It shouldn't be that slow. Y.
[1]
# add_cluster: Success (in 0:00:03) # add_hosts: * Collect artifacts: - [Thread-5] lago-basic-suite-4-1-host1: ERROR (in 0:00:19) - [Thread-4] lago-basic-suite-4-1-engine: ERROR (in 0:00:20) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout
Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths ignore_nopath=ignore_nopath, File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp propagate_fail=False File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from local_path=local_path, File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get self._recv_all() File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all msg = self.channel.recv(1024) File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv raise socket.timeout() timeout * Collect artifacts: ERROR (in 0:00:33)
# add_hosts: ERROR (in 0:15:52) # Collect artifacts: * [Thread-19] lago-basic-suite-4-1-engine: * [Thread-20] lago-basic-suite-4-1-host1: * [Thread-21] lago-basic-suite-4-1-host0: * [Thread-21] lago-basic-suite-4-1-host0: Success (in 0:00:03) * [Thread-20] lago-basic-suite-4-1-host1: Success (in 0:00:05) * [Thread-19] lago-basic-suite-4-1-engine: Success (in 0:00:16) # Collect artifacts: Success (in 0:00:16) # Results located at /home/myoung/repos/github/ovir t-system-tests/deployment-basic-suite-4.1/default/002_bootst rap.py.junit.xml @ Run test: 002_bootstrap.py: ERROR (in 0:16:57) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run self.cli_plugins[args.ovirtverb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper return func(*args, prefix=prefix, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest raise RuntimeError('Some tests failed') RuntimeError: Some tests failed + res=1 + cd - /home/myoung/repos/github/ovirt-system-tests + return 1 + failed=true + env_collect /home/myoung/repos/github/ovir t-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py + local tests_out_dir=/home/myoung/repos/github/ovirt-system-tests/ test_logs/basic-suite-4.1/post-002_bootstrap.py + echo '#########################' ######################### + [[ -e /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1 ]] + cd /home/myoung/repos/github/ovirt-system-tests/deployment-basi c-suite-4.1/current + lago collect --output /home/myoung/repos/github/ovir t-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py @ Collect artifacts: # [Thread-1] lago-basic-suite-4-1-engine: # [Thread-2] lago-basic-suite-4-1-host1: # [Thread-3] lago-basic-suite-4-1-host0: # [Thread-3] lago-basic-suite-4-1-host0: Success (in 0:00:02) # [Thread-2] lago-basic-suite-4-1-host1: Success (in 0:00:03) # [Thread-1] lago-basic-suite-4-1-engine: Success (in 0:00:06) @ Collect artifacts: Success (in 0:00:07) + cp -a logs /home/myoung/repos/github/ovirt-system-tests/test_logs/basic -suite-4.1/post-002_bootstrap.py/lago_logs + cd - /home/myoung/repos/github/ovirt-system-tests + true + echo '@@@@ ERROR: Failed running /home/myoung/repos/github/ovir t-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py' @@@@ ERROR: Failed running /home/myoung/repos/github/ovir t-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py + return 1
[2]
$ cat './test_logs/basic-suite-4.1/post-002_bootstrap.py.2017-08-14_21:55:00/lago_logs/lago.log' | fpaste Uploading (1202.8KiB)... https://da.gd/6aw2 -> https://paste.fedoraproject.or g/paste/956xJBRY-m1D9T7giQktHQ/
[3]
$ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log | grep ovirt-engine); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine- setup-20170814212716-ql8ekq.log Uploading (5544.7KiB)... https://da.gd/QLkSD -> https://paste.fedoraproject.or g/paste/tQEEAfL3TQk7nYhrdyQIVQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.5KiB)... https://da.gd/eX4c6 -> https://paste.fedoraproject.or g/paste/7YFWv6uVzARJ8q0iCAhUNQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (33.9KiB)... https://da.gd/n1JTd -> https://paste.fedoraproject.or g/paste/~cSehNxN77eGxprxliE3dQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2697.0KiB)... https://da.gd/pRDI -> https://paste.fedoraproject.or g/paste/-jJF5TRug~pcQ7syEK~qnA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/7mkNW -> https://paste.fedoraproject.or g/paste/PjqtJIsVK~1UzHd1PtjUJA/ ----
On Mon, Aug 14, 2017 at 2:02 PM, Marc Young <3vilpenguin@gmail.com> wrote:
rpm -q python-ovirt-engine-sdk4
Mine is outdated:
$ rpm -q python-ovirt-engine-sdk4 python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64
Ill update that and run again.
Logs (better to send too many than not enough sometimes). Ovirt engine logs are there.
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲ $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log); do echo $i; cat $i | fpaste; echo ----; done test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/tuned/tuned.log Uploading (3.5KiB)... https://da.gd/LiG0 -> https://paste.fedoraproject.or g/paste/wed7pYARTmHHTHcf0DvVAQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/audit/audit.log Uploading (401.2KiB)... https://da.gd/tgSw -> https://paste.fedoraproject.or g/paste/WEqegOFiLpiNGCv0hvY0ww/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/yum.log Uploading (24.3KiB)... https://da.gd/BzoMg -> https://paste.fedoraproject.or g/paste/qla7z~hRsym8kkCr6NlXFg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/setup-ds.log Uploading (37.4KiB)... https://da.gd/1HHKA -> https://paste.fedoraproject.or g/paste/4nRRDoC8lrv5CTxbS6TCxQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-set up-20170814074645-vzih31.log Uploading (5544.4KiB)... https://da.gd/xYpYl -> https://paste.fedoraproject.or g/paste/goz7RQVvusLXjiiheLD5xg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/boot.log Uploading (1.3KiB)... https://da.gd/4Y6pE -> https://paste.fedoraproject.or g/paste/OCqWe6tjGkrNOrTkqex3mg/ ----
test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/server.log Uploading (34.1KiB)... https://da.gd/PZm7d -> https://paste.fedoraproject.or g/paste/HpZOi8LTrgcWhKqfPeowPA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine/engine.log Uploading (2724.0KiB)... https://da.gd/pIOO -> https://paste.fedoraproject.or g/paste/ufbtnfZOfSmun8sbOCs1xg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log Uploading (2.4KiB)... https://da.gd/Xf5R -> https://paste.fedoraproject.or g/paste/MO-dlfNQXi19Tla3T~QXsA/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-engine/_var_log/snmptrapd.log Uploading (0.1KiB)... https://da.gd/cF48 -> https://paste.fedoraproject.or g/paste/NYqECCL1QVihcBQV~VWDsQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/G5oC -> https://paste.fedoraproject.or g/paste/JgvhOLoA7cEO2Q3ESjP5Aw/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/audit/audit.log Uploading (183.1KiB)... https://da.gd/9bjYn -> https://paste.fedoraproject.or g/paste/Ado1joqnZxkPL1us6UeOwQ/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host1/_var_log/yum.log Uploading (18.3KiB)... https://da.gd/i5ki -> https://paste.fedoraproject.or g/paste/9sxQziGLRqw1tLHKfBAi6Q/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/tuned/tuned.log Uploading (1.5KiB)... https://da.gd/5Iiu4 -> https://paste.fedoraproject.or g/paste/RaqsqKQdfgcTP1o2Epo~Pg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/audit/audit.log Uploading (179.9KiB)... https://da.gd/MyZ7 -> https://paste.fedoraproject.or g/paste/x7T7ZkIS3g5JE2dj028AVg/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s uite-4-1-host0/_var_log/yum.log Uploading (17.4KiB)... https://da.gd/Gmux -> https://paste.fedoraproject.or g/paste/kORmR4ShCv1EVGtTWSu2Zw/ ---- test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs/lago.log Uploading (1198.1KiB)... https://da.gd/vzVCA -> https://paste.fedoraproject.or g/paste/TuQiMEro23I~OnFcM7wG-g/ ----
On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri@redhat.com> wrote:
On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin@gmail.com> wrote:
You should have all logs needed @/home/myoung/repos/github/ovi
rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap .py/lago_logs
The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. Heres all logs for basic suite 4.1 that I know how to find:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
The engine and hosts logs are usually under the test_logs, which should be in the same subdir as lago_logs directory. Also, can you verify you have the latest python v4 SDK installled?
rpm -q python-ovirt-engine-sdk4
If you don't have it installed, you can get it directly from the build job ( which acts also as yum repo ):
el7 build - http://jenkins.ovirt.org/job/p ython-ovirt-engine-sdk4_4.1_build-artifacts-el7-x86_64/lastS uccessfulBuild/artifact/exported-artifacts/
fc26 build is available only for master [1] but should also work.
There are also fc24/25 builds, just follow the same pattern to find them on Jenkins.
[1] http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_ma ster_build-artifacts-fc26-x86_64/lastSuccessfulBuild/artifac t/exported-artifacts/
$ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat $i/*.log | fpaste; done Uploading (1355.0KiB)... https://da.gd/OIlR -> https://paste.fedoraproject.or g/paste/9Xn2RuDgF2pY2I0CouBbGg/ Uploading (1154.0KiB)... https://da.gd/0Cti -> https://paste.fedoraproject.or g/paste/uJqvAiDSblnmN8OA6i5tXA/ Uploading (1377.8KiB)... https://da.gd/GpDIf -> https://paste.fedoraproject.or g/paste/~Cn6IJgbvateJfFulDNFCg/ Uploading (1176.7KiB)... https://da.gd/skhBy -> https://paste.fedoraproject.or g/paste/HIpEVm4TRPpH5nJCOyUfWA/ Uploading (1397.1KiB)... https://da.gd/fEtF -> https://paste.fedoraproject.or g/paste/cuycP~u9D~gJKNPsQ5NH2g/ Uploading (1198.1KiB)... https://da.gd/1zNm -> https://paste.fedoraproject.or g/paste/5~yEDwCSr~tdxHT~XUaLPg/
On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:
On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:
> Full log is just as uninteresting: https://paste.f > edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA >
I was actually hoping for the engine log. Y.
> > On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> > wrote: > >> >> >> On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> >> wrote: >> >>> Thanks for clarifying, that makes sense in hindsight since its >>> testing things in their entirety. This is more of a rabbit hole for me as >>> Im really just trying to learn Lago by using the OST project (since I'll be >>> duplicating a lot of the setup). >>> >>> stdout logs from a run: https://pastebin.com/KBDaCCYp >>> >> >> Can you send engine log so we'll understand why the host >> installation failed? >> You should have all logs needed @/home/myoung/repos/github/ovi >> rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap >> .py/lago_logs >> >> Y. >> >> >>> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> >>> wrote: >>> >>>> >>>> >>>> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young < >>>> 3vilpenguin@gmail.com> wrote: >>>> >>>>> Ill try to get some detailed log files later, but fwiw I'm not >>>>> running the hc suites (afaik, Im still getting inundated with the system >>>>> tests/lago). The link I used for Jenkins was just to try to pull the latest >>>>> 'passing' hash for ovirt-system-tests >>>>> >>>> >>>> I see, its a bit more complicated than that I'm afraid, the HASH >>>> you see is probably on for OST code itself, while the suites can fail on >>>> any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so >>>> its more than just a single hash, but rather a list of RPMs and their >>>> versions, together with hash of OST ( for the tests themselves ) and also >>>> OS updates. >>>> >>>> >>>>> >>>>> The errors have all been from master branch on the basic suite >>>>> for 4.1 via: >>>>> >>>>> $ ./run_suite.sh basic-suite-4.1/ >>>>> >>>>> >>>> OK, then we need to investigate it and understand why add host >>>> fails on 4.1, please provide logs when possible. >>>> >>>> BTW, you can also try running the manual job [1], but you'll need >>>> to provide it with custom yum repo URL with your built artifacts to test >>>> your code. >>>> There is a section on it on the OST readthedocs page. >>>> >>>> [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job >>>> /ovirt-system-tests_manual/ >>>> >>>> >>>>> >>>>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> >>>>> wrote: >>>>> >>>>>> I see you run the 'hc' suite, which means hyperconverged. This >>>>>> suite is running an hosted engine on gluster storage, It's a more complex >>>>>> suite than the basic one, and prone to more errors, however it should still >>>>>> work, but if you don't require it specifically, I would recommend running >>>>>> the basic suite, which should be easier to debug and also quicker to run. >>>>>> >>>>>> I'm also adding hc maintainer so she can check, can you share >>>>>> the link to the Jenkins job you're running? Or the log files if you're >>>>>> running locally. >>>>>> >>>>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Actually I spoke too soon, still fails: >>>>>>> >>>>>>> + lago ovirt runtest /home/myoung/repos/github/ovir >>>>>>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py >>>>>>> @ Run test: 002_bootstrap.py: >>>>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_', >>>>>>> '^setup\\.py$'] >>>>>>> # print_api_ver: >>>>>>> # print_api_ver: Success (in 0:00:00) >>>>>>> # add_dc: >>>>>>> # add_dc: Success (in 0:00:43) >>>>>>> # add_cluster: >>>>>>> # add_cluster: Success (in 0:00:03) >>>>>>> # add_hosts: >>>>>>> dd >>>>>>> * Collect artifacts: >>>>>>> * Collect artifacts: Success (in 0:01:14) >>>>>>> # add_hosts: Success (in 0:16:36) >>>>>>> # Results located at /home/myoung/repos/github/ovir >>>>>>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j >>>>>>> unit.xml >>>>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26) >>>>>>> Error occured, aborting >>>>>>> Traceback (most recent call last): >>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", >>>>>>> line 360, in do_run >>>>>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", >>>>>>> line 184, in do_run >>>>>>> self._do_run(**vars(args)) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>>> 501, in wrapper >>>>>>> return func(*args, **kwargs) >>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>>> 512, in wrapper >>>>>>> return func(*args, prefix=prefix, **kwargs) >>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", >>>>>>> line 99, in do_ovirt_runtest >>>>>>> raise RuntimeError('Some tests failed') >>>>>>> RuntimeError: Some tests failed >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young < >>>>>>> 3vilpenguin@gmail.com> wrote: >>>>>>> >>>>>>>> Edit: reply-all >>>>>>>> >>>>>>>> It does, every time. >>>>>>>> I got it to pass by using the last passing revision from >>>>>>>> Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), >>>>>>>> noted from http://jenkins.ovirt.org/job/s >>>>>>>> ystem-tests_hc-basic-suite-master/15/consoleFull >>>>>>>> >>>>>>>> Not sure if it's a true race condition, that build in Jenkins >>>>>>>> has failed consistently since #15 >>>>>>>> >>>>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young < >>>>>>>>> 3vilpenguin@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> What's the fix for this for those of us using >>>>>>>>>> ovirt-system-tests? >>>>>>>>>> >>>>>>>>>> Im trying to adapt some of the code for testing a third >>>>>>>>>> party tool, but master is still failing on 002_bootstrap >>>>>>>>>> >>>>>>>>> >>>>>>>>> It fails consistently? >>>>>>>>> AFAIK this is a race condition that happens maybe once a >>>>>>>>> week, can you share your logs? is it also failing on add_secondary_storage? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik < >>>>>>>>>> bzlotnik@redhat.com> wrote: >>>>>>>>>> >>>>>>>>>>> I'm not sure it's related since the LSM test runs much >>>>>>>>>>> later, in 004 >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri < >>>>>>>>>>> eedri@redhat.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Adding Allon & Benny. >>>>>>>>>>>> >>>>>>>>>>>> Is it possible to verify this is related to the LSM issue >>>>>>>>>>>> we've been handling in [1]? >>>>>>>>>>>> If this is the case, we agreed to disable the test next >>>>>>>>>>>> time it fails, as the current workarounds with sleep isn't enough. >>>>>>>>>>>> >>>>>>>>>>>> Can you confirm this is the case, and so we'll have to >>>>>>>>>>>> disable this test until one of the RFEs described in [1] is merged? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/ >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren < >>>>>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer < >>>>>>>>>>>>> nsoffer@redhat.com> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren < >>>>>>>>>>>>> bkorren@redhat.com> wrote: >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains >>>>>>>>>>>>> ] >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Link to suspected patches: >>>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974 >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > This patch adds missing log when resizing an online >>>>>>>>>>>>> disk - why do you think >>>>>>>>>>>>> > it is related to the failure? >>>>>>>>>>>>> >>>>>>>>>>>>> Because it is the only patch participating in the test. >>>>>>>>>>>>> (The test was equivalent to running the manual job with >>>>>>>>>>>>> just this patch) >>>>>>>>>>>>> >>>>>>>>>>>>> Then again this may also one of the usual SD testing >>>>>>>>>>>>> race conditions. >>>>>>>>>>>>> Since the code in the patch seemed to be related to >>>>>>>>>>>>> storage I didn't >>>>>>>>>>>>> want to just assume that. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Barak Korren >>>>>>>>>>>>> RHV DevOps team , RHCE, RHCi >>>>>>>>>>>>> Red Hat EMEA >>>>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | >>>>>>>>>>>>> redhat.com/trusted >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Devel mailing list >>>>>>>>>>>>> Devel@ovirt.org >>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Eyal edri >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ASSOCIATE MANAGER >>>>>>>>>>>> >>>>>>>>>>>> RHV DevOps >>>>>>>>>>>> >>>>>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>>>>> <https://redhat.com/trusted> >>>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list >>>>>>>>>>> Devel@ovirt.org >>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Eyal edri >>>>>>>>> >>>>>>>>> >>>>>>>>> ASSOCIATE MANAGER >>>>>>>>> >>>>>>>>> RHV DevOps >>>>>>>>> >>>>>>>>> EMEA VIRTUALIZATION R&D >>>>>>>>> >>>>>>>>> >>>>>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>>>>> <https://redhat.com/trusted> >>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018> >>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Eyal edri >>>> >>>> >>>> ASSOCIATE MANAGER >>>> >>>> RHV DevOps >>>> >>>> EMEA VIRTUALIZATION R&D >>>> >>>> >>>> Red Hat EMEA <https://www.redhat.com/> >>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>> <https://redhat.com/trusted> >>>> phone: +972-9-7692018 <+972%209-769-2018> >>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>> >>> >>> >>> _______________________________________________ >>> Devel mailing list >>> Devel@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/devel >>> >> >> >
--
Eyal edri
ASSOCIATE MANAGER
RHV DevOps
EMEA VIRTUALIZATION R&D
Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 <+972%209-769-2018> irc: eedri (on #tlv #rhev-dev #rhev-integ)
participants (7)
-
Barak Korren -
Benny Zlotnik -
Eyal Edri -
Marc Young -
Nadav Goldin -
Nir Soffer -
Yaniv Kaul