[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 01/08/2017 ] [add_secondary_storage_domains]
Marc Young
3vilpenguin at gmail.com
Tue Aug 15 12:05:17 UTC 2017
Its in KVM. Ill bump the specs. Thanks for bearing with me, I'll see what I
can do. It wasnt immediately obvious
On Tue, Aug 15, 2017 at 4:24 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>
>
> On Tue, Aug 15, 2017 at 5:28 AM, Marc Young <3vilpenguin at gmail.com> wrote:
>
>> After updating the python sdk:
>>
>> myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
>> $ rpm -q python-ovirt-engine-sdk4
>> python-ovirt-engine-sdk4-4.1.6-2.20170712git1b99f36.el7.centos.x86_64
>>
>>
>> I get more but different errors[1].
>>
>> Then the lago log[2]
>> All engine Logs are even further[3]
>>
>
> OK, understood the issue. Your host installation is for some reason very
> very slow - the test just times out after 15 minutes. It's why the Engine
> log is a bit cut.
> It begins here:
> 2017-08-14 *21:38:32*,143-04 DEBUG [org.ovirt.engine.core.uutils.ssh.SSHClient]
> (default task-10) [10728bdc-0cdc-45ba-90f8-7a78b0b79d6e] Connecting
> 'root at lago-basic-suite-4-1-host0'
>
> And it still installing here:
> 2017-08-14 *21:55:01*,438-04 INFO [org.ovirt.engine.core.dal.
> dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [76cb1954]
> EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Correlation ID: 76cb1954, Call
> Stack: null, Custom Event ID: -1, Message: Installing Host
> lago-basic-suite-4-1-host0. Yum install: 239/364:
> python-netaddr-0.7.5-7.el7.noarch.
>
> But ovirt-system-tests already given up. On my laptop it takes ~3 minutes
> - perhaps you are running on qemu and not kvm, or something similar? It
> shouldn't be that slow.
> Y.
>
>
>
>>
>> [1]
>>
>> # add_cluster: Success (in 0:00:03)
>> # add_hosts:
>> * Collect artifacts:
>> - [Thread-5] lago-basic-suite-4-1-host1: ERROR (in 0:00:19)
>> - [Thread-4] lago-basic-suite-4-1-engine: ERROR (in 0:00:20)
>> Error while running thread
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
>> _ret_via_queue
>> queue.put({'return': func()})
>> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in
>> _collect_artifacts
>> vm.collect_artifacts(path, ignore_nopath)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624,
>> in collect_artifacts
>> ignore_nopath=ignore_nopath
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381,
>> in extract_paths
>> return self.provider.extract_paths(paths, *args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py",
>> line 342, in extract_paths
>> ignore_nopath=ignore_nopath,
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247,
>> in extract_paths
>> self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266,
>> in _extract_paths_scp
>> propagate_fail=False
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425,
>> in copy_from
>> local_path=local_path,
>> File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
>> self._recv_all()
>> File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
>> msg = self.channel.recv(1024)
>> File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615,
>> in recv
>> raise socket.timeout()
>> timeout
>> Error while running thread
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
>> _ret_via_queue
>> queue.put({'return': func()})
>> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in
>> _collect_artifacts
>> vm.collect_artifacts(path, ignore_nopath)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624,
>> in collect_artifacts
>> ignore_nopath=ignore_nopath
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381,
>> in extract_paths
>> return self.provider.extract_paths(paths, *args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py",
>> line 342, in extract_paths
>> ignore_nopath=ignore_nopath,
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247,
>> in extract_paths
>> self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266,
>> in _extract_paths_scp
>> propagate_fail=False
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425,
>> in copy_from
>> local_path=local_path,
>> File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
>> self._recv_all()
>> File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
>> msg = self.channel.recv(1024)
>> File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615,
>> in recv
>> raise socket.timeout()
>> timeout
>>
>> Error while running thread
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
>> _ret_via_queue
>> queue.put({'return': func()})
>> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in
>> _collect_artifacts
>> vm.collect_artifacts(path, ignore_nopath)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624,
>> in collect_artifacts
>> ignore_nopath=ignore_nopath
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381,
>> in extract_paths
>> return self.provider.extract_paths(paths, *args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py",
>> line 342, in extract_paths
>> ignore_nopath=ignore_nopath,
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247,
>> in extract_paths
>> self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266,
>> in _extract_paths_scp
>> propagate_fail=False
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425,
>> in copy_from
>> local_path=local_path,
>> File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
>> self._recv_all()
>> File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
>> msg = self.channel.recv(1024)
>> File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615,
>> in recv
>> raise socket.timeout()
>> timeout
>> Error while running thread
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
>> _ret_via_queue
>> queue.put({'return': func()})
>> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in
>> _collect_artifacts
>> vm.collect_artifacts(path, ignore_nopath)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624,
>> in collect_artifacts
>> ignore_nopath=ignore_nopath
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381,
>> in extract_paths
>> return self.provider.extract_paths(paths, *args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py",
>> line 342, in extract_paths
>> ignore_nopath=ignore_nopath,
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247,
>> in extract_paths
>> self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266,
>> in _extract_paths_scp
>> propagate_fail=False
>> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425,
>> in copy_from
>> local_path=local_path,
>> File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
>> self._recv_all()
>> File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
>> msg = self.channel.recv(1024)
>> File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615,
>> in recv
>> raise socket.timeout()
>> timeout
>> * Collect artifacts: ERROR (in 0:00:33)
>>
>> # add_hosts: ERROR (in 0:15:52)
>> # Collect artifacts:
>> * [Thread-19] lago-basic-suite-4-1-engine:
>> * [Thread-20] lago-basic-suite-4-1-host1:
>> * [Thread-21] lago-basic-suite-4-1-host0:
>> * [Thread-21] lago-basic-suite-4-1-host0: Success (in 0:00:03)
>> * [Thread-20] lago-basic-suite-4-1-host1: Success (in 0:00:05)
>> * [Thread-19] lago-basic-suite-4-1-engine: Success (in 0:00:16)
>> # Collect artifacts: Success (in 0:00:16)
>> # Results located at /home/myoung/repos/github/ovir
>> t-system-tests/deployment-basic-suite-4.1/default/002_bootst
>> rap.py.junit.xml
>> @ Run test: 002_bootstrap.py: ERROR (in 0:16:57)
>> Error occured, aborting
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in
>> do_run
>> self.cli_plugins[args.ovirtverb].do_run(args)
>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184,
>> in do_run
>> self._do_run(**vars(args))
>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in
>> wrapper
>> return func(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in
>> wrapper
>> return func(*args, prefix=prefix, **kwargs)
>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in
>> do_ovirt_runtest
>> raise RuntimeError('Some tests failed')
>> RuntimeError: Some tests failed
>> + res=1
>> + cd -
>> /home/myoung/repos/github/ovirt-system-tests
>> + return 1
>> + failed=true
>> + env_collect /home/myoung/repos/github/ovir
>> t-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py
>> + local tests_out_dir=/home/myoung/repos/github/ovirt-system-tests/
>> test_logs/basic-suite-4.1/post-002_bootstrap.py
>> + echo '#########################'
>> #########################
>> + [[ -e /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1
>> ]]
>> + cd /home/myoung/repos/github/ovirt-system-tests/deployment-basi
>> c-suite-4.1/current
>> + lago collect --output /home/myoung/repos/github/ovir
>> t-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py
>> @ Collect artifacts:
>> # [Thread-1] lago-basic-suite-4-1-engine:
>> # [Thread-2] lago-basic-suite-4-1-host1:
>> # [Thread-3] lago-basic-suite-4-1-host0:
>> # [Thread-3] lago-basic-suite-4-1-host0: Success (in 0:00:02)
>> # [Thread-2] lago-basic-suite-4-1-host1: Success (in 0:00:03)
>> # [Thread-1] lago-basic-suite-4-1-engine: Success (in 0:00:06)
>> @ Collect artifacts: Success (in 0:00:07)
>> + cp -a logs /home/myoung/repos/github/ovirt-system-tests/test_logs/basic
>> -suite-4.1/post-002_bootstrap.py/lago_logs
>> + cd -
>> /home/myoung/repos/github/ovirt-system-tests
>> + true
>> + echo '@@@@ ERROR: Failed running /home/myoung/repos/github/ovir
>> t-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py'
>> @@@@ ERROR: Failed running /home/myoung/repos/github/ovir
>> t-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py
>> + return 1
>>
>> [2]
>>
>> $ cat './test_logs/basic-suite-4.1/post-002_bootstrap.py.2017-08-14_21:55:00/lago_logs/lago.log'
>> | fpaste
>> Uploading (1202.8KiB)...
>> https://da.gd/6aw2 -> https://paste.fedoraproject.or
>> g/paste/956xJBRY-m1D9T7giQktHQ/
>>
>> [3]
>>
>> $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name
>> *.log | grep ovirt-engine); do echo $i; cat $i | fpaste; echo ----; done
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>> uite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-
>> setup-20170814212716-ql8ekq.log
>> Uploading (5544.7KiB)...
>> https://da.gd/QLkSD -> https://paste.fedoraproject.or
>> g/paste/tQEEAfL3TQk7nYhrdyQIVQ/
>> ----
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>> uite-4-1-engine/_var_log/ovirt-engine/boot.log
>> Uploading (1.5KiB)...
>> https://da.gd/eX4c6 -> https://paste.fedoraproject.or
>> g/paste/7YFWv6uVzARJ8q0iCAhUNQ/
>> ----
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>> uite-4-1-engine/_var_log/ovirt-engine/server.log
>> Uploading (33.9KiB)...
>> https://da.gd/n1JTd -> https://paste.fedoraproject.or
>> g/paste/~cSehNxN77eGxprxliE3dQ/
>> ----
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>> uite-4-1-engine/_var_log/ovirt-engine/engine.log
>> Uploading (2697.0KiB)...
>> https://da.gd/pRDI -> https://paste.fedoraproject.or
>> g/paste/-jJF5TRug~pcQ7syEK~qnA/
>> ----
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>> uite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log
>> Uploading (2.4KiB)...
>> https://da.gd/7mkNW -> https://paste.fedoraproject.or
>> g/paste/PjqtJIsVK~1UzHd1PtjUJA/
>> ----
>>
>>
>> On Mon, Aug 14, 2017 at 2:02 PM, Marc Young <3vilpenguin at gmail.com>
>> wrote:
>>
>>> rpm -q python-ovirt-engine-sdk4
>>>>
>>>
>>> Mine is outdated:
>>>
>>> $ rpm -q python-ovirt-engine-sdk4
>>> python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64
>>>
>>> Ill update that and run again.
>>>
>>>
>>>
>>> Logs (better to send too many than not enough sometimes). Ovirt engine
>>> logs are there.
>>>
>>> myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
>>> $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name
>>> *.log); do echo $i; cat $i | fpaste; echo ----; done
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/tuned/tuned.log
>>> Uploading (3.5KiB)...
>>> https://da.gd/LiG0 -> https://paste.fedoraproject.or
>>> g/paste/wed7pYARTmHHTHcf0DvVAQ/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/audit/audit.log
>>> Uploading (401.2KiB)...
>>> https://da.gd/tgSw -> https://paste.fedoraproject.or
>>> g/paste/WEqegOFiLpiNGCv0hvY0ww/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/yum.log
>>> Uploading (24.3KiB)...
>>> https://da.gd/BzoMg -> https://paste.fedoraproject.or
>>> g/paste/qla7z~hRsym8kkCr6NlXFg/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/setup-ds.log
>>> Uploading (37.4KiB)...
>>> https://da.gd/1HHKA -> https://paste.fedoraproject.or
>>> g/paste/4nRRDoC8lrv5CTxbS6TCxQ/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-set
>>> up-20170814074645-vzih31.log
>>> Uploading (5544.4KiB)...
>>> https://da.gd/xYpYl -> https://paste.fedoraproject.or
>>> g/paste/goz7RQVvusLXjiiheLD5xg/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/ovirt-engine/boot.log
>>> Uploading (1.3KiB)...
>>> https://da.gd/4Y6pE -> https://paste.fedoraproject.or
>>> g/paste/OCqWe6tjGkrNOrTkqex3mg/
>>> ----
>>>
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/ovirt-engine/server.log
>>> Uploading (34.1KiB)...
>>> https://da.gd/PZm7d -> https://paste.fedoraproject.or
>>> g/paste/HpZOi8LTrgcWhKqfPeowPA/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/ovirt-engine/engine.log
>>> Uploading (2724.0KiB)...
>>> https://da.gd/pIOO -> https://paste.fedoraproject.or
>>> g/paste/ufbtnfZOfSmun8sbOCs1xg/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log
>>> Uploading (2.4KiB)...
>>> https://da.gd/Xf5R -> https://paste.fedoraproject.or
>>> g/paste/MO-dlfNQXi19Tla3T~QXsA/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-engine/_var_log/snmptrapd.log
>>> Uploading (0.1KiB)...
>>> https://da.gd/cF48 -> https://paste.fedoraproject.or
>>> g/paste/NYqECCL1QVihcBQV~VWDsQ/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-host1/_var_log/tuned/tuned.log
>>> Uploading (1.5KiB)...
>>> https://da.gd/G5oC -> https://paste.fedoraproject.or
>>> g/paste/JgvhOLoA7cEO2Q3ESjP5Aw/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-host1/_var_log/audit/audit.log
>>> Uploading (183.1KiB)...
>>> https://da.gd/9bjYn -> https://paste.fedoraproject.or
>>> g/paste/Ado1joqnZxkPL1us6UeOwQ/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-host1/_var_log/yum.log
>>> Uploading (18.3KiB)...
>>> https://da.gd/i5ki -> https://paste.fedoraproject.or
>>> g/paste/9sxQziGLRqw1tLHKfBAi6Q/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-host0/_var_log/tuned/tuned.log
>>> Uploading (1.5KiB)...
>>> https://da.gd/5Iiu4 -> https://paste.fedoraproject.or
>>> g/paste/RaqsqKQdfgcTP1o2Epo~Pg/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-host0/_var_log/audit/audit.log
>>> Uploading (179.9KiB)...
>>> https://da.gd/MyZ7 -> https://paste.fedoraproject.or
>>> g/paste/x7T7ZkIS3g5JE2dj028AVg/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-s
>>> uite-4-1-host0/_var_log/yum.log
>>> Uploading (17.4KiB)...
>>> https://da.gd/Gmux -> https://paste.fedoraproject.or
>>> g/paste/kORmR4ShCv1EVGtTWSu2Zw/
>>> ----
>>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs/lago.log
>>> Uploading (1198.1KiB)...
>>> https://da.gd/vzVCA -> https://paste.fedoraproject.or
>>> g/paste/TuQiMEro23I~OnFcM7wG-g/
>>> ----
>>>
>>>
>>> On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin at gmail.com>
>>>> wrote:
>>>>
>>>>> You should have all logs needed @/home/myoung/repos/github/ovi
>>>>>> rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap
>>>>>> .py/lago_logs
>>>>>
>>>>>
>>>>> The one I pasted into fpaste was the only one in that directory, if
>>>>> there are more logs Im just unsure where they live.
>>>>> Heres all logs for basic suite 4.1 that I know how to find:
>>>>>
>>>>> myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
>>>>>
>>>>>
>>>> The engine and hosts logs are usually under the test_logs, which should
>>>> be in the same subdir as lago_logs directory.
>>>> Also, can you verify you have the latest python v4 SDK installled?
>>>>
>>>> rpm -q python-ovirt-engine-sdk4
>>>>
>>>> If you don't have it installed, you can get it directly from the build
>>>> job ( which acts also as yum repo ):
>>>>
>>>> el7 build - http://jenkins.ovirt.org/job/p
>>>> ython-ovirt-engine-sdk4_4.1_build-artifacts-el7-x86_64/lastS
>>>> uccessfulBuild/artifact/exported-artifacts/
>>>>
>>>> fc26 build is available only for master [1] but should also work.
>>>>
>>>> There are also fc24/25 builds, just follow the same pattern to find
>>>> them on Jenkins.
>>>>
>>>>
>>>> [1] http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_ma
>>>> ster_build-artifacts-fc26-x86_64/lastSuccessfulBuild/artifac
>>>> t/exported-artifacts/
>>>>
>>>>
>>>>
>>>>> $ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat
>>>>> $i/*.log | fpaste; done
>>>>> Uploading (1355.0KiB)...
>>>>> https://da.gd/OIlR -> https://paste.fedoraproject.or
>>>>> g/paste/9Xn2RuDgF2pY2I0CouBbGg/
>>>>> Uploading (1154.0KiB)...
>>>>> https://da.gd/0Cti -> https://paste.fedoraproject.or
>>>>> g/paste/uJqvAiDSblnmN8OA6i5tXA/
>>>>> Uploading (1377.8KiB)...
>>>>> https://da.gd/GpDIf -> https://paste.fedoraproject.or
>>>>> g/paste/~Cn6IJgbvateJfFulDNFCg/
>>>>> Uploading (1176.7KiB)...
>>>>> https://da.gd/skhBy -> https://paste.fedoraproject.or
>>>>> g/paste/HIpEVm4TRPpH5nJCOyUfWA/
>>>>> Uploading (1397.1KiB)...
>>>>> https://da.gd/fEtF -> https://paste.fedoraproject.or
>>>>> g/paste/cuycP~u9D~gJKNPsQ5NH2g/
>>>>> Uploading (1198.1KiB)...
>>>>> https://da.gd/1zNm -> https://paste.fedoraproject.or
>>>>> g/paste/5~yEDwCSr~tdxHT~XUaLPg/
>>>>>
>>>>>
>>>>> On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Full log is just as uninteresting: https://paste.f
>>>>>>> edoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
>>>>>>>
>>>>>>
>>>>>> I was actually hoping for the engine log.
>>>>>> Y.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for clarifying, that makes sense in hindsight since its
>>>>>>>>> testing things in their entirety. This is more of a rabbit hole for me as
>>>>>>>>> Im really just trying to learn Lago by using the OST project (since I'll be
>>>>>>>>> duplicating a lot of the setup).
>>>>>>>>>
>>>>>>>>> stdout logs from a run: https://pastebin.com/KBDaCCYp
>>>>>>>>>
>>>>>>>>
>>>>>>>> Can you send engine log so we'll understand why the host
>>>>>>>> installation failed?
>>>>>>>> You should have all logs needed @/home/myoung/repos/github/ovi
>>>>>>>> rt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap
>>>>>>>> .py/lago_logs
>>>>>>>>
>>>>>>>> Y.
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri at redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <
>>>>>>>>>> 3vilpenguin at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ill try to get some detailed log files later, but fwiw I'm not
>>>>>>>>>>> running the hc suites (afaik, Im still getting inundated with the system
>>>>>>>>>>> tests/lago). The link I used for Jenkins was just to try to pull the latest
>>>>>>>>>>> 'passing' hash for ovirt-system-tests
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I see, its a bit more complicated than that I'm afraid, the HASH
>>>>>>>>>> you see is probably on for OST code itself, while the suites can fail on
>>>>>>>>>> any oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so
>>>>>>>>>> its more than just a single hash, but rather a list of RPMs and their
>>>>>>>>>> versions, together with hash of OST ( for the tests themselves ) and also
>>>>>>>>>> OS updates.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The errors have all been from master branch on the basic suite
>>>>>>>>>>> for 4.1 via:
>>>>>>>>>>>
>>>>>>>>>>> $ ./run_suite.sh basic-suite-4.1/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> OK, then we need to investigate it and understand why add host
>>>>>>>>>> fails on 4.1, please provide logs when possible.
>>>>>>>>>>
>>>>>>>>>> BTW, you can also try running the manual job [1], but you'll need
>>>>>>>>>> to provide it with custom yum repo URL with your built artifacts to test
>>>>>>>>>> your code.
>>>>>>>>>> There is a section on it on the OST readthedocs page.
>>>>>>>>>>
>>>>>>>>>> [1] http://jenkins.ovirt.org/view/oVirt%20system%20tests/job
>>>>>>>>>> /ovirt-system-tests_manual/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri at redhat.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I see you run the 'hc' suite, which means hyperconverged. This
>>>>>>>>>>>> suite is running an hosted engine on gluster storage, It's a more complex
>>>>>>>>>>>> suite than the basic one, and prone to more errors, however it should still
>>>>>>>>>>>> work, but if you don't require it specifically, I would recommend running
>>>>>>>>>>>> the basic suite, which should be easier to debug and also quicker to run.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm also adding hc maintainer so she can check, can you share
>>>>>>>>>>>> the link to the Jenkins job you're running? Or the log files if you're
>>>>>>>>>>>> running locally.
>>>>>>>>>>>>
>>>>>>>>>>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Actually I spoke too soon, still fails:
>>>>>>>>>>>>>
>>>>>>>>>>>>> + lago ovirt runtest /home/myoung/repos/github/ovir
>>>>>>>>>>>>> t-system-tests/vagrant/test-scenarios/002_bootstrap.py
>>>>>>>>>>>>> @ Run test: 002_bootstrap.py:
>>>>>>>>>>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_',
>>>>>>>>>>>>> '^setup\\.py$']
>>>>>>>>>>>>> # print_api_ver:
>>>>>>>>>>>>> # print_api_ver: Success (in 0:00:00)
>>>>>>>>>>>>> # add_dc:
>>>>>>>>>>>>> # add_dc: Success (in 0:00:43)
>>>>>>>>>>>>> # add_cluster:
>>>>>>>>>>>>> # add_cluster: Success (in 0:00:03)
>>>>>>>>>>>>> # add_hosts:
>>>>>>>>>>>>> dd
>>>>>>>>>>>>> * Collect artifacts:
>>>>>>>>>>>>> * Collect artifacts: Success (in 0:01:14)
>>>>>>>>>>>>> # add_hosts: Success (in 0:16:36)
>>>>>>>>>>>>> # Results located at /home/myoung/repos/github/ovir
>>>>>>>>>>>>> t-system-tests/deployment-vagrant/default/002_bootstrap.py.j
>>>>>>>>>>>>> unit.xml
>>>>>>>>>>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26)
>>>>>>>>>>>>> Error occured, aborting
>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py",
>>>>>>>>>>>>> line 360, in do_run
>>>>>>>>>>>>> self.cli_plugins[args.ovirtverb].do_run(args)
>>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py",
>>>>>>>>>>>>> line 184, in do_run
>>>>>>>>>>>>> self._do_run(**vars(args))
>>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line
>>>>>>>>>>>>> 501, in wrapper
>>>>>>>>>>>>> return func(*args, **kwargs)
>>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line
>>>>>>>>>>>>> 512, in wrapper
>>>>>>>>>>>>> return func(*args, prefix=prefix, **kwargs)
>>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py",
>>>>>>>>>>>>> line 99, in do_ovirt_runtest
>>>>>>>>>>>>> raise RuntimeError('Some tests failed')
>>>>>>>>>>>>> RuntimeError: Some tests failed
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young <
>>>>>>>>>>>>> 3vilpenguin at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Edit: reply-all
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It does, every time.
>>>>>>>>>>>>>> I got it to pass by using the last passing revision from
>>>>>>>>>>>>>> Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ),
>>>>>>>>>>>>>> noted from http://jenkins.ovirt.org/job/s
>>>>>>>>>>>>>> ystem-tests_hc-basic-suite-master/15/consoleFull
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Not sure if it's a true race condition, that build in Jenkins
>>>>>>>>>>>>>> has failed consistently since #15
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri at redhat.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young <
>>>>>>>>>>>>>>> 3vilpenguin at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What's the fix for this for those of us using
>>>>>>>>>>>>>>>> ovirt-system-tests?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Im trying to adapt some of the code for testing a third
>>>>>>>>>>>>>>>> party tool, but master is still failing on 002_bootstrap
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It fails consistently?
>>>>>>>>>>>>>>> AFAIK this is a race condition that happens maybe once a
>>>>>>>>>>>>>>> week, can you share your logs? is it also failing on add_secondary_storage?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik <
>>>>>>>>>>>>>>>> bzlotnik at redhat.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm not sure it's related since the LSM test runs much
>>>>>>>>>>>>>>>>> later, in 004
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <
>>>>>>>>>>>>>>>>> eedri at redhat.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Adding Allon & Benny.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is it possible to verify this is related to the LSM issue
>>>>>>>>>>>>>>>>>> we've been handling in [1]?
>>>>>>>>>>>>>>>>>> If this is the case, we agreed to disable the test next
>>>>>>>>>>>>>>>>>> time it fails, as the current workarounds with sleep isn't enough.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you confirm this is the case, and so we'll have to
>>>>>>>>>>>>>>>>>> disable this test until one of the RFEs described in [1] is merged?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren <
>>>>>>>>>>>>>>>>>> bkorren at redhat.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <
>>>>>>>>>>>>>>>>>>> nsoffer at redhat.com> wrote:
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren <
>>>>>>>>>>>>>>>>>>> bkorren at redhat.com> wrote:
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> Test failed: [ 002_bootstrap.add_secondary_storage_domains
>>>>>>>>>>>>>>>>>>> ]
>>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>>> >> Link to suspected patches:
>>>>>>>>>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>> > This patch adds missing log when resizing an online
>>>>>>>>>>>>>>>>>>> disk - why do you think
>>>>>>>>>>>>>>>>>>> > it is related to the failure?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Because it is the only patch participating in the test.
>>>>>>>>>>>>>>>>>>> (The test was equivalent to running the manual job with
>>>>>>>>>>>>>>>>>>> just this patch)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Then again this may also one of the usual SD testing
>>>>>>>>>>>>>>>>>>> race conditions.
>>>>>>>>>>>>>>>>>>> Since the code in the patch seemed to be related to
>>>>>>>>>>>>>>>>>>> storage I didn't
>>>>>>>>>>>>>>>>>>> want to just assume that.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Barak Korren
>>>>>>>>>>>>>>>>>>> RHV DevOps team , RHCE, RHCi
>>>>>>>>>>>>>>>>>>> Red Hat EMEA
>>>>>>>>>>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. |
>>>>>>>>>>>>>>>>>>> redhat.com/trusted
>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>> Devel mailing list
>>>>>>>>>>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Eyal edri
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ASSOCIATE MANAGER
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> RHV DevOps
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>>>>>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>>>>>>>>>>>>>> <https://redhat.com/trusted>
>>>>>>>>>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>>>>>>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Devel mailing list
>>>>>>>>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Eyal edri
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ASSOCIATE MANAGER
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> RHV DevOps
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>>>>>>>>>>> <https://redhat.com/trusted>
>>>>>>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>>>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Eyal edri
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ASSOCIATE MANAGER
>>>>>>>>>>
>>>>>>>>>> RHV DevOps
>>>>>>>>>>
>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Red Hat EMEA <https://www.redhat.com/>
>>>>>>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>>>>>>>> <https://redhat.com/trusted>
>>>>>>>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Devel mailing list
>>>>>>>>> Devel at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Eyal edri
>>>>
>>>>
>>>> ASSOCIATE MANAGER
>>>>
>>>> RHV DevOps
>>>>
>>>> EMEA VIRTUALIZATION R&D
>>>>
>>>>
>>>> Red Hat EMEA <https://www.redhat.com/>
>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED.
>>>> <https://redhat.com/trusted>
>>>> phone: +972-9-7692018 <+972%209-769-2018>
>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170815/8568910d/attachment-0001.html>
More information about the Devel
mailing list