Its in KVM. Ill bump the specs. Thanks for bearing with me, I'll see what I can do. It wasnt immediately obvious

On Tue, Aug 15, 2017 at 4:24 AM, Yaniv Kaul <ykaul@redhat.com> wrote:


On Tue, Aug 15, 2017 at 5:28 AM, Marc Young <3vilpenguin@gmail.com> wrote:
After updating the python sdk:
myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
$ rpm -q python-ovirt-engine-sdk4
python-ovirt-engine-sdk4-4.1.6-2.20170712git1b99f36.el7.centos.x86_64

I get more but different errors[1].

Then the lago log[2]
All engine Logs are even further[3]

OK, understood the issue. Your host installation is for some reason very very slow - the test just times out after 15 minutes. It's why the Engine log is a bit cut.
It begins here:
2017-08-14 21:38:32,143-04 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient] (default task-10) [10728bdc-0cdc-45ba-90f8-7a78b0b79d6e] Connecting 'root@lago-basic-suite-4-1-host0'

And it still installing here:
2017-08-14 21:55:01,438-04 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [76cb1954] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Correlation ID: 76cb1954, Call Stack: null, Custom Event ID: -1, Message: Installing Host lago-basic-suite-4-1-host0. Yum install: 239/364: python-netaddr-0.7.5-7.el7.noarch.

But ovirt-system-tests already given up. On my laptop it takes ~3 minutes - perhaps you are running on qemu and not kvm, or something similar? It shouldn't be that slow.
Y.

 

[1]
  # add_cluster: Success (in 0:00:03)
  # add_hosts:
    * Collect artifacts:
      - [Thread-5] lago-basic-suite-4-1-host1: ERROR (in 0:00:19)
      - [Thread-4] lago-basic-suite-4-1-engine: ERROR (in 0:00:20)
Error while running thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts
    vm.collect_artifacts(path, ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts
    ignore_nopath=ignore_nopath
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths
    return self.provider.extract_paths(paths, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths
    ignore_nopath=ignore_nopath,
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths
    self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp
    propagate_fail=False
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from
    local_path=local_path,
  File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
    self._recv_all()
  File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
    msg = self.channel.recv(1024)
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv
    raise socket.timeout()
timeout
Error while running thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts
    vm.collect_artifacts(path, ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts
    ignore_nopath=ignore_nopath
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths
    return self.provider.extract_paths(paths, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths
    ignore_nopath=ignore_nopath,
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths
    self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp
    propagate_fail=False
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from
    local_path=local_path,
  File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
    self._recv_all()
  File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
    msg = self.channel.recv(1024)
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv
    raise socket.timeout()
timeout

Error while running thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts
    vm.collect_artifacts(path, ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts
    ignore_nopath=ignore_nopath
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths
    return self.provider.extract_paths(paths, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths
    ignore_nopath=ignore_nopath,
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths
    self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp
    propagate_fail=False
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from
    local_path=local_path,
  File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
    self._recv_all()
  File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
    msg = self.channel.recv(1024)
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv
    raise socket.timeout()
timeout
Error while running thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in _collect_artifacts
    vm.collect_artifacts(path, ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in collect_artifacts
    ignore_nopath=ignore_nopath
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in extract_paths
    return self.provider.extract_paths(paths, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line 342, in extract_paths
    ignore_nopath=ignore_nopath,
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in extract_paths
    self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in _extract_paths_scp
    propagate_fail=False
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in copy_from
    local_path=local_path,
  File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
    self._recv_all()
  File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
    msg = self.channel.recv(1024)
  File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in recv
    raise socket.timeout()
timeout
    * Collect artifacts: ERROR (in 0:00:33)

  # add_hosts: ERROR (in 0:15:52)
  # Collect artifacts:
    * [Thread-19] lago-basic-suite-4-1-engine:
    * [Thread-20] lago-basic-suite-4-1-host1:
    * [Thread-21] lago-basic-suite-4-1-host0:
    * [Thread-21] lago-basic-suite-4-1-host0: Success (in 0:00:03)
    * [Thread-20] lago-basic-suite-4-1-host1: Success (in 0:00:05)
    * [Thread-19] lago-basic-suite-4-1-engine: Success (in 0:00:16)
  # Collect artifacts: Success (in 0:00:16)
  # Results located at /home/myoung/repos/github/ovirt-system-tests/deployment-basic-suite-4.1/default/002_bootstrap.py.junit.xml
@ Run test: 002_bootstrap.py: ERROR (in 0:16:57)
Error occured, aborting
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run
    self.cli_plugins[args.ovirtverb].do_run(args)
  File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run
    self._do_run(**vars(args))
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper
    return func(*args, prefix=prefix, **kwargs)
  File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest
    raise RuntimeError('Some tests failed')
RuntimeError: Some tests failed
+ res=1
+ cd -
/home/myoung/repos/github/ovirt-system-tests
+ return 1
+ failed=true
+ env_collect /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py
+ local tests_out_dir=/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py
+ echo '#########################'
#########################
+ [[ -e /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1 ]]
+ cd /home/myoung/repos/github/ovirt-system-tests/deployment-basic-suite-4.1/current
+ lago collect --output /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py
@ Collect artifacts:
  # [Thread-1] lago-basic-suite-4-1-engine:
  # [Thread-2] lago-basic-suite-4-1-host1:
  # [Thread-3] lago-basic-suite-4-1-host0:
  # [Thread-3] lago-basic-suite-4-1-host0: Success (in 0:00:02)
  # [Thread-2] lago-basic-suite-4-1-host1: Success (in 0:00:03)
  # [Thread-1] lago-basic-suite-4-1-engine: Success (in 0:00:06)
@ Collect artifacts: Success (in 0:00:07)
+ cp -a logs /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs
+ cd -
/home/myoung/repos/github/ovirt-system-tests
+ true
+ echo '@@@@ ERROR: Failed running /home/myoung/repos/github/ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py'
@@@@ ERROR: Failed running /home/myoung/repos/github/ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py
+ return 1

[2]
$ cat './test_logs/basic-suite-4.1/post-002_bootstrap.py.2017-08-14_21:55:00/lago_logs/lago.log' | fpaste
Uploading (1202.8KiB)...

[3]
$ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log | grep ovirt-engine); do echo $i; cat $i | fpaste; echo ----; done
Uploading (5544.7KiB)...
----
Uploading (1.5KiB)...
----
Uploading (33.9KiB)...
----
Uploading (2697.0KiB)...
----
Uploading (2.4KiB)...
----

On Mon, Aug 14, 2017 at 2:02 PM, Marc Young <3vilpenguin@gmail.com> wrote:
rpm -q python-ovirt-engine-sdk4
 
Mine is outdated: 

$ rpm -q python-ovirt-engine-sdk4
python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64

Ill update that and run again.
 
Logs (better to send too many than not enough sometimes). Ovirt engine logs are there.

myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
$ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name *.log); do echo $i; cat $i | fpaste; echo ----; done
Uploading (3.5KiB)...
----
Uploading (401.2KiB)...
----
Uploading (24.3KiB)...
----
Uploading (37.4KiB)...
----
Uploading (5544.4KiB)...
----
Uploading (1.3KiB)...
----
Uploading (34.1KiB)...
----
Uploading (2724.0KiB)...
----
Uploading (2.4KiB)...
----
Uploading (0.1KiB)...
----
Uploading (1.5KiB)...
----
Uploading (183.1KiB)...
----
Uploading (18.3KiB)...
----
Uploading (1.5KiB)...
----
Uploading (179.9KiB)...
----
Uploading (17.4KiB)...
----
Uploading (1198.1KiB)...
----

On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri@redhat.com> wrote:


On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin@gmail.com> wrote:
You should have all logs needed @/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs

The one I pasted into fpaste was the only one in that directory, if there are more logs Im just unsure where they live. 
Heres all logs for basic suite 4.1 that I know how to find:

myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲

The engine and hosts logs are usually under the test_logs, which should be in the same subdir as lago_logs directory.
Also, can you verify you have the latest python v4 SDK installled?

rpm -q python-ovirt-engine-sdk4

If you don't have it installed, you can get it directly from the build job ( which acts also as yum repo ):


fc26 build is available only for master [1] but should also work.

There are also fc24/25 builds, just follow the same pattern to find them on Jenkins.



 

On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul@redhat.com> wrote:


On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin@gmail.com> wrote:

I was actually hoping for the engine log.
Y.
 

On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul@redhat.com> wrote:


On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Thanks for clarifying, that makes sense in hindsight since its testing things in their entirety. This is more of a rabbit hole for me as Im really just trying to learn Lago by using the OST project (since I'll be duplicating a lot of the setup).

stdout logs from a run: https://pastebin.com/KBDaCCYp

Can you send engine log so we'll understand why the host installation failed?
You should have all logs needed @/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs

Y.


On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri@redhat.com> wrote:


On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Ill try to get some detailed log files later, but fwiw I'm not running the hc suites (afaik, Im still getting inundated with the system tests/lago). The link I used for Jenkins was just to try to pull the latest 'passing' hash for ovirt-system-tests

I see, its a bit more complicated than that I'm afraid, the HASH you see is probably on for OST code itself, while the suites can fail on any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than just a single hash, but rather a list of RPMs and their versions, together with hash of OST ( for the tests themselves ) and also OS updates. 
 

The errors have all been from master branch on the basic suite for 4.1 via:

$ ./run_suite.sh basic-suite-4.1/

OK, then we need to investigate it and understand why add host fails on 4.1, please provide logs when possible.

BTW, you can also try running the manual job [1], but you'll need to provide it with custom yum repo URL with your built artifacts to test your code.
There is a section on it on the OST readthedocs page.

 

On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri@redhat.com> wrote:
I see you run the 'hc' suite, which means hyperconverged. This suite is running an hosted engine on gluster storage, It's a more complex suite than the basic one, and prone to more errors, however it should still work, but if you don't require it specifically, I would recommend running the basic suite, which should be easier to debug and also quicker to run.

I'm also adding hc maintainer so she can check, can you share the link to the Jenkins job you're running? Or the log files if you're running locally.

On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin@gmail.com> wrote:
Actually I spoke too soon, still fails:

+ lago ovirt runtest /home/myoung/repos/github/ovirt-system-tests/vagrant/test-scenarios/002_bootstrap.py
@ Run test: 002_bootstrap.py:
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
  # print_api_ver:
  # print_api_ver: Success (in 0:00:00)
  # add_dc:
  # add_dc: Success (in 0:00:43)
  # add_cluster:
  # add_cluster: Success (in 0:00:03)
  # add_hosts:
dd
    * Collect artifacts:
    * Collect artifacts: Success (in 0:01:14)
  # add_hosts: Success (in 0:16:36)
  # Results located at /home/myoung/repos/github/ovirt-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml
@ Run test: 002_bootstrap.py: Success (in 0:17:26)
Error occured, aborting
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in do_run
    self.cli_plugins[args.ovirtverb].do_run(args)
  File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run
    self._do_run(**vars(args))
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in wrapper
    return func(*args, prefix=prefix, **kwargs)
  File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest
    raise RuntimeError('Some tests failed')
RuntimeError: Some tests failed




On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <3vilpenguin@gmail.com> wrote:
Edit: reply-all

It does, every time.
I got it to pass by using the last passing revision from Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-master/15/consoleFull

Not sure if it's a true race condition, that build in Jenkins has failed consistently since #15

On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri@redhat.com> wrote:


On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <3vilpenguin@gmail.com> wrote:
What's the fix for this for those of us using ovirt-system-tests?

Im trying to adapt some of the code for testing a third party tool, but master is still failing on 002_bootstrap

It fails consistently? 
AFAIK this is a race condition that happens maybe once a week, can you share your logs? is it also failing on add_secondary_storage? 

 

On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <bzlotnik@redhat.com> wrote:
I'm not sure it's related since the LSM test runs much later, in 004

On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri@redhat.com> wrote:
Adding Allon & Benny.

Is it possible to verify this is related to the LSM issue we've been handling in [1]? 
If this is the case, we agreed to disable the test next time it fails, as the current workarounds with sleep isn't enough.

Can you confirm this is the case, and so we'll have to disable this test until one of the RFEs described in [1] is merged?



On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <bkorren@redhat.com> wrote:
On 1 August 2017 at 14:39, Nir Soffer <nsoffer@redhat.com> wrote:
>
>
> On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <bkorren@redhat.com> wrote:
>>
>> Test failed: [ 002_bootstrap.add_secondary_storage_domains ]
>>
>> Link to suspected patches:
>> https://gerrit.ovirt.org/#/c/79974
>
>
> This patch adds missing log when resizing an online disk - why do you think
> it is related to the failure?

Because it is the only patch participating in the test.
(The test was equivalent to running the manual job with just this patch)

Then again this may also one of the usual SD testing race conditions.
Since the code in the patch seemed to be related to storage I didn't
want to just assume that.


--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel





--

Eyal edri


ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D


Red Hat EMEA

TRIED. TESTED. TRUSTED.
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)


_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel




--

Eyal edri


ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D


Red Hat EMEA

TRIED. TESTED. TRUSTED.
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)






--

Eyal edri


ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D


Red Hat EMEA

TRIED. TESTED. TRUSTED.
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)


_______________________________________________
Devel mailing list
Devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel







--

Eyal edri


ASSOCIATE MANAGER

RHV DevOps

EMEA VIRTUALIZATION R&D


Red Hat EMEA

TRIED. TESTED. TRUSTED.
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)