Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).

stdout logs from a run: https://pastebin.com/KBDaCCYp

On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:


On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests

I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates. 
 

The errors have all been from master branch on the basic suite for 4.1 via:

$ ./run_suite.sh basic-suite-4.1/

OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.

BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code.
There is a section on it on the OST readthedocs page.

 

On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.

I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.

On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
Actually I spoke too soon, still fails:

+ lago ovirt runtest /home/myoung/repos/github/ovirt-system-tests/vagrant/test-scenarios/002_bootstrap.py
@ Run test: 002_bootstrap.py:
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
  # print_api_ver:
  # print_api_ver: Success (in 0:00:00)
  # add_dc:
  # add_dc: Success (in 0:00:43)
  # add_cluster:
  # add_cluster: Success (in 0:00:03)
  # add_hosts:
dd
    * Collect artifacts:
    * Collect artifacts: Success (in 0:01:14)
  # add_hosts: Success (in 0:16:36)
  # Results located at /home/myoung/repos/github/ovirt-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml
@ Run test: 002_bootstrap.py: Success (in 0:17:26)
Error occured, aborting
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run
    self.cli_plugins[args.ovirtverb].do_run(args)
  File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run
    self._do_run(**vars(args))
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper
    return func(*args, prefix=prefix, **kwargs)
  File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest
    raise RuntimeError('Some tests failed')
RuntimeError: Some tests failed




On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Edit: reply-all

It does, every time.
I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-master/15/consoleFull

Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15

On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:


On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
What's the fix for this for those of us using ovirt-system-tests?

Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap

It fails consistently? 
AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage? 

 

On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
I'm not sure it's related since the LSM test runs much later, in 004

On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
Adding Allon & Benny.

Is it possible to verify this is related to the LSM issue we've been handling in [1]? 
If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.

Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?



On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
>
>
> On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> wrote:
>>
>> Test failed: [ 002_bootstrap.add_secondary_storage_domains ]
>>
>> Link to suspected patches:
>> https://gerrit.ovirt.org/#/c/79974
>
>
> This patch adds missing log when resizing an online disk - why do you think
> it is related to the failure?

Because it is the only patch participating in the test.
(The test was equivalent to running the manual job with just this patch)

Then again this may also one of the usual SD testing race conditions.
Since the code in the patch seemed to be related to storage I didn't
want to just assume that.


--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel





--

Eyal edri


ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D


Red Hat EMEA

TRIED. TESTED. TRUSTED.
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)


_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel




--

Eyal edri


ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D


Red Hat EMEA

TRIED. TESTED. TRUSTED.
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)






--

Eyal edri


ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D


Red Hat EMEA

TRIED. TESTED. TRUSTED.
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)