[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 01/08/2017 ] [add_secondary_storage_domains]
Nadav Goldin
ngoldin at redhat.com
Tue Aug 15 07:54:03 UTC 2017
Hi Marc,
Some of the logs failed to extract, so I'm not sure what went wrong.
Specifically, host-deploy directory and vdms logs from the hosts.
(They should all be under test_logs, I would just tar.gz the entire
directory and send it next time). As "add hosts" step failed on
timeout(15 minutes) - I suspect is due to a missing package or network
issue(some packages might be downloaded from the VDMS hosts during
that stage). So:
1. Could you retry? is this consistent?
2. If so, can you please upload the entire log directory?
If we discover this indeed is a network issue, you could just increase
the timeout.
Thanks,
Nadav.
On Tue, Aug 15, 2017 at 5:28 AM, Marc Young <3vilpenguin at gmail.com> wrote:
> After updating the python sdk:
>
> myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
> $ rpm -q python-ovirt-engine-sdk4
> python-ovirt-engine-sdk4-4.1.6-2.20170712git1b99f36.el7.centos.x86_64
>
>
> I get more but different errors[1].
>
> Then the lago log[2]
> All engine Logs are even further[3]
>
> [1]
>
> # add_cluster: Success (in 0:00:03)
> # add_hosts:
> * Collect artifacts:
> - [Thread-5] lago-basic-suite-4-1-host1: ERROR (in 0:00:19)
> - [Thread-4] lago-basic-suite-4-1-engine: ERROR (in 0:00:20)
> Error while running thread
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
> _ret_via_queue
> queue.put({'return': func()})
> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in
> _collect_artifacts
> vm.collect_artifacts(path, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in
> collect_artifacts
> ignore_nopath=ignore_nopath
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in
> extract_paths
> return self.provider.extract_paths(paths, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line
> 342, in extract_paths
> ignore_nopath=ignore_nopath,
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in
> extract_paths
> self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in
> _extract_paths_scp
> propagate_fail=False
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in
> copy_from
> local_path=local_path,
> File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
> self._recv_all()
> File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
> msg = self.channel.recv(1024)
> File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in
> recv
> raise socket.timeout()
> timeout
> Error while running thread
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
> _ret_via_queue
> queue.put({'return': func()})
> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in
> _collect_artifacts
> vm.collect_artifacts(path, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in
> collect_artifacts
> ignore_nopath=ignore_nopath
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in
> extract_paths
> return self.provider.extract_paths(paths, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line
> 342, in extract_paths
> ignore_nopath=ignore_nopath,
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in
> extract_paths
> self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in
> _extract_paths_scp
> propagate_fail=False
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in
> copy_from
> local_path=local_path,
> File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
> self._recv_all()
> File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
> msg = self.channel.recv(1024)
> File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in
> recv
> raise socket.timeout()
> timeout
>
> Error while running thread
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
> _ret_via_queue
> queue.put({'return': func()})
> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in
> _collect_artifacts
> vm.collect_artifacts(path, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in
> collect_artifacts
> ignore_nopath=ignore_nopath
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in
> extract_paths
> return self.provider.extract_paths(paths, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line
> 342, in extract_paths
> ignore_nopath=ignore_nopath,
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in
> extract_paths
> self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in
> _extract_paths_scp
> propagate_fail=False
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in
> copy_from
> local_path=local_path,
> File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
> self._recv_all()
> File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
> msg = self.channel.recv(1024)
> File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in
> recv
> raise socket.timeout()
> timeout
> Error while running thread
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
> _ret_via_queue
> queue.put({'return': func()})
> File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1478, in
> _collect_artifacts
> vm.collect_artifacts(path, ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 624, in
> collect_artifacts
> ignore_nopath=ignore_nopath
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 381, in
> extract_paths
> return self.provider.extract_paths(paths, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/providers/libvirt/vm.py", line
> 342, in extract_paths
> ignore_nopath=ignore_nopath,
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 247, in
> extract_paths
> self._extract_paths_scp(paths=paths, ignore_nopath=ignore_nopath)
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 266, in
> _extract_paths_scp
> propagate_fail=False
> File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 425, in
> copy_from
> local_path=local_path,
> File "/usr/lib/python2.7/site-packages/scp.py", line 125, in get
> self._recv_all()
> File "/usr/lib/python2.7/site-packages/scp.py", line 250, in _recv_all
> msg = self.channel.recv(1024)
> File "/usr/lib/python2.7/site-packages/paramiko/channel.py", line 615, in
> recv
> raise socket.timeout()
> timeout
> * Collect artifacts: ERROR (in 0:00:33)
>
> # add_hosts: ERROR (in 0:15:52)
> # Collect artifacts:
> * [Thread-19] lago-basic-suite-4-1-engine:
> * [Thread-20] lago-basic-suite-4-1-host1:
> * [Thread-21] lago-basic-suite-4-1-host0:
> * [Thread-21] lago-basic-suite-4-1-host0: Success (in 0:00:03)
> * [Thread-20] lago-basic-suite-4-1-host1: Success (in 0:00:05)
> * [Thread-19] lago-basic-suite-4-1-engine: Success (in 0:00:16)
> # Collect artifacts: Success (in 0:00:16)
> # Results located at
> /home/myoung/repos/github/ovirt-system-tests/deployment-basic-suite-4.1/default/002_bootstrap.py.junit.xml
> @ Run test: 002_bootstrap.py: ERROR (in 0:16:57)
> Error occured, aborting
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 360, in
> do_run
> self.cli_plugins[args.ovirtverb].do_run(args)
> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in
> do_run
> self._do_run(**vars(args))
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 501, in
> wrapper
> return func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 512, in
> wrapper
> return func(*args, prefix=prefix, **kwargs)
> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in
> do_ovirt_runtest
> raise RuntimeError('Some tests failed')
> RuntimeError: Some tests failed
> + res=1
> + cd -
> /home/myoung/repos/github/ovirt-system-tests
> + return 1
> + failed=true
> + env_collect
> /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py
> + local
> tests_out_dir=/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py
> + echo '#########################'
> #########################
> + [[ -e
> /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1 ]]
> + cd
> /home/myoung/repos/github/ovirt-system-tests/deployment-basic-suite-4.1/current
> + lago collect --output
> /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py
> @ Collect artifacts:
> # [Thread-1] lago-basic-suite-4-1-engine:
> # [Thread-2] lago-basic-suite-4-1-host1:
> # [Thread-3] lago-basic-suite-4-1-host0:
> # [Thread-3] lago-basic-suite-4-1-host0: Success (in 0:00:02)
> # [Thread-2] lago-basic-suite-4-1-host1: Success (in 0:00:03)
> # [Thread-1] lago-basic-suite-4-1-engine: Success (in 0:00:06)
> @ Collect artifacts: Success (in 0:00:07)
> + cp -a logs
> /home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs
> + cd -
> /home/myoung/repos/github/ovirt-system-tests
> + true
> + echo '@@@@ ERROR: Failed running
> /home/myoung/repos/github/ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py'
> @@@@ ERROR: Failed running
> /home/myoung/repos/github/ovirt-system-tests/basic-suite-4.1/test-scenarios/002_bootstrap.py
> + return 1
>
> [2]
>
> $ cat
> './test_logs/basic-suite-4.1/post-002_bootstrap.py.2017-08-14_21:55:00/lago_logs/lago.log'
> | fpaste
> Uploading (1202.8KiB)...
> https://da.gd/6aw2 ->
> https://paste.fedoraproject.org/paste/956xJBRY-m1D9T7giQktHQ/
>
> [3]
>
> $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name
> *.log | grep ovirt-engine); do echo $i; cat $i | fpaste; echo ----; done
> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20170814212716-ql8ekq.log
> Uploading (5544.7KiB)...
> https://da.gd/QLkSD ->
> https://paste.fedoraproject.org/paste/tQEEAfL3TQk7nYhrdyQIVQ/
> ----
> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/boot.log
> Uploading (1.5KiB)...
> https://da.gd/eX4c6 ->
> https://paste.fedoraproject.org/paste/7YFWv6uVzARJ8q0iCAhUNQ/
> ----
> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/server.log
> Uploading (33.9KiB)...
> https://da.gd/n1JTd ->
> https://paste.fedoraproject.org/paste/~cSehNxN77eGxprxliE3dQ/
> ----
> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/engine.log
> Uploading (2697.0KiB)...
> https://da.gd/pRDI ->
> https://paste.fedoraproject.org/paste/-jJF5TRug~pcQ7syEK~qnA/
> ----
> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log
> Uploading (2.4KiB)...
> https://da.gd/7mkNW ->
> https://paste.fedoraproject.org/paste/PjqtJIsVK~1UzHd1PtjUJA/
> ----
>
>
> On Mon, Aug 14, 2017 at 2:02 PM, Marc Young <3vilpenguin at gmail.com> wrote:
>>>
>>> rpm -q python-ovirt-engine-sdk4
>>
>>
>> Mine is outdated:
>>
>> $ rpm -q python-ovirt-engine-sdk4
>> python-ovirt-engine-sdk4-4.1.5-2.el7.centos.x86_64
>>
>> Ill update that and run again.
>>
>>
>>
>> Logs (better to send too many than not enough sometimes). Ovirt engine
>> logs are there.
>>
>> myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
>> $ for i in $(find test_logs/basic-suite-4.1/post-002_bootstrap.py -name
>> *.log); do echo $i; cat $i | fpaste; echo ----; done
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/tuned/tuned.log
>> Uploading (3.5KiB)...
>> https://da.gd/LiG0 ->
>> https://paste.fedoraproject.org/paste/wed7pYARTmHHTHcf0DvVAQ/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/audit/audit.log
>> Uploading (401.2KiB)...
>> https://da.gd/tgSw ->
>> https://paste.fedoraproject.org/paste/WEqegOFiLpiNGCv0hvY0ww/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/yum.log
>> Uploading (24.3KiB)...
>> https://da.gd/BzoMg ->
>> https://paste.fedoraproject.org/paste/qla7z~hRsym8kkCr6NlXFg/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/setup-ds.log
>> Uploading (37.4KiB)...
>> https://da.gd/1HHKA ->
>> https://paste.fedoraproject.org/paste/4nRRDoC8lrv5CTxbS6TCxQ/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/setup/ovirt-engine-setup-20170814074645-vzih31.log
>> Uploading (5544.4KiB)...
>> https://da.gd/xYpYl ->
>> https://paste.fedoraproject.org/paste/goz7RQVvusLXjiiheLD5xg/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/boot.log
>> Uploading (1.3KiB)...
>> https://da.gd/4Y6pE ->
>> https://paste.fedoraproject.org/paste/OCqWe6tjGkrNOrTkqex3mg/
>> ----
>>
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/server.log
>> Uploading (34.1KiB)...
>> https://da.gd/PZm7d ->
>> https://paste.fedoraproject.org/paste/HpZOi8LTrgcWhKqfPeowPA/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/engine.log
>> Uploading (2724.0KiB)...
>> https://da.gd/pIOO ->
>> https://paste.fedoraproject.org/paste/ufbtnfZOfSmun8sbOCs1xg/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine-dwh/ovirt-engine-dwhd.log
>> Uploading (2.4KiB)...
>> https://da.gd/Xf5R ->
>> https://paste.fedoraproject.org/paste/MO-dlfNQXi19Tla3T~QXsA/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-engine/_var_log/snmptrapd.log
>> Uploading (0.1KiB)...
>> https://da.gd/cF48 ->
>> https://paste.fedoraproject.org/paste/NYqECCL1QVihcBQV~VWDsQ/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/tuned/tuned.log
>> Uploading (1.5KiB)...
>> https://da.gd/G5oC ->
>> https://paste.fedoraproject.org/paste/JgvhOLoA7cEO2Q3ESjP5Aw/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/audit/audit.log
>> Uploading (183.1KiB)...
>> https://da.gd/9bjYn ->
>> https://paste.fedoraproject.org/paste/Ado1joqnZxkPL1us6UeOwQ/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host1/_var_log/yum.log
>> Uploading (18.3KiB)...
>> https://da.gd/i5ki ->
>> https://paste.fedoraproject.org/paste/9sxQziGLRqw1tLHKfBAi6Q/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/tuned/tuned.log
>> Uploading (1.5KiB)...
>> https://da.gd/5Iiu4 ->
>> https://paste.fedoraproject.org/paste/RaqsqKQdfgcTP1o2Epo~Pg/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/audit/audit.log
>> Uploading (179.9KiB)...
>> https://da.gd/MyZ7 ->
>> https://paste.fedoraproject.org/paste/x7T7ZkIS3g5JE2dj028AVg/
>> ----
>>
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago-basic-suite-4-1-host0/_var_log/yum.log
>> Uploading (17.4KiB)...
>> https://da.gd/Gmux ->
>> https://paste.fedoraproject.org/paste/kORmR4ShCv1EVGtTWSu2Zw/
>> ----
>> test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs/lago.log
>> Uploading (1198.1KiB)...
>> https://da.gd/vzVCA ->
>> https://paste.fedoraproject.org/paste/TuQiMEro23I~OnFcM7wG-g/
>> ----
>>
>>
>> On Mon, Aug 14, 2017 at 1:11 PM, Eyal Edri <eedri at redhat.com> wrote:
>>>
>>>
>>>
>>> On Mon, Aug 14, 2017 at 7:18 PM, Marc Young <3vilpenguin at gmail.com>
>>> wrote:
>>>>>
>>>>> You should have all logs needed
>>>>> @/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs
>>>>
>>>>
>>>> The one I pasted into fpaste was the only one in that directory, if
>>>> there are more logs Im just unsure where they live.
>>>> Heres all logs for basic suite 4.1 that I know how to find:
>>>>
>>>> myoung at dev-vm in ~/repos/github/ovirt-system-tests on (no branch)▲
>>>
>>>
>>> The engine and hosts logs are usually under the test_logs, which should
>>> be in the same subdir as lago_logs directory.
>>> Also, can you verify you have the latest python v4 SDK installled?
>>>
>>> rpm -q python-ovirt-engine-sdk4
>>>
>>> If you don't have it installed, you can get it directly from the build
>>> job ( which acts also as yum repo ):
>>>
>>> el7 build -
>>> http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_4.1_build-artifacts-el7-x86_64/lastSuccessfulBuild/artifact/exported-artifacts/
>>>
>>> fc26 build is available only for master [1] but should also work.
>>>
>>> There are also fc24/25 builds, just follow the same pattern to find them
>>> on Jenkins.
>>>
>>>
>>> [1]
>>> http://jenkins.ovirt.org/job/python-ovirt-engine-sdk4_master_build-artifacts-fc26-x86_64/lastSuccessfulBuild/artifact/exported-artifacts/
>>>
>>>
>>>>
>>>> $ for i in $(find . -name 'lago_logs' | grep basic-suite-4.1); do cat
>>>> $i/*.log | fpaste; done
>>>> Uploading (1355.0KiB)...
>>>> https://da.gd/OIlR ->
>>>> https://paste.fedoraproject.org/paste/9Xn2RuDgF2pY2I0CouBbGg/
>>>> Uploading (1154.0KiB)...
>>>> https://da.gd/0Cti ->
>>>> https://paste.fedoraproject.org/paste/uJqvAiDSblnmN8OA6i5tXA/
>>>> Uploading (1377.8KiB)...
>>>> https://da.gd/GpDIf ->
>>>> https://paste.fedoraproject.org/paste/~Cn6IJgbvateJfFulDNFCg/
>>>> Uploading (1176.7KiB)...
>>>> https://da.gd/skhBy ->
>>>> https://paste.fedoraproject.org/paste/HIpEVm4TRPpH5nJCOyUfWA/
>>>> Uploading (1397.1KiB)...
>>>> https://da.gd/fEtF ->
>>>> https://paste.fedoraproject.org/paste/cuycP~u9D~gJKNPsQ5NH2g/
>>>> Uploading (1198.1KiB)...
>>>> https://da.gd/1zNm ->
>>>> https://paste.fedoraproject.org/paste/5~yEDwCSr~tdxHT~XUaLPg/
>>>>
>>>>
>>>> On Mon, Aug 14, 2017 at 10:14 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 14, 2017 at 5:10 PM, Marc Young <3vilpenguin at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Full log is just as uninteresting:
>>>>>> https://paste.fedoraproject.org/paste/ZaZMcT5sg9SAGQ7OKC3-EA
>>>>>
>>>>>
>>>>> I was actually hoping for the engine log.
>>>>> Y.
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 14, 2017 at 8:03 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Aug 14, 2017 at 3:31 PM, Marc Young <3vilpenguin at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Thanks for clarifying, that makes sense in hindsight since its
>>>>>>>> testing things in their entirety. This is more of a rabbit hole for me as Im
>>>>>>>> really just trying to learn Lago by using the OST project (since I'll be
>>>>>>>> duplicating a lot of the setup).
>>>>>>>>
>>>>>>>> stdout logs from a run: https://pastebin.com/KBDaCCYp
>>>>>>>
>>>>>>>
>>>>>>> Can you send engine log so we'll understand why the host installation
>>>>>>> failed?
>>>>>>> You should have all logs needed
>>>>>>> @/home/myoung/repos/github/ovirt-system-tests/test_logs/basic-suite-4.1/post-002_bootstrap.py/lago_logs
>>>>>>>
>>>>>>> Y.
>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 14, 2017 at 6:44 AM, Eyal Edri <eedri at redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 14, 2017 at 2:31 PM, Marc Young <3vilpenguin at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Ill try to get some detailed log files later, but fwiw I'm not
>>>>>>>>>> running the hc suites (afaik, Im still getting inundated with the system
>>>>>>>>>> tests/lago). The link I used for Jenkins was just to try to pull the latest
>>>>>>>>>> 'passing' hash for ovirt-system-tests
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I see, its a bit more complicated than that I'm afraid, the HASH
>>>>>>>>> you see is probably on for OST code itself, while the suites can fail on any
>>>>>>>>> oVirt project ( ovirt-engine/vdsm/host-deploy/etc.... ), so its more than
>>>>>>>>> just a single hash, but rather a list of RPMs and their versions, together
>>>>>>>>> with hash of OST ( for the tests themselves ) and also OS updates.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The errors have all been from master branch on the basic suite for
>>>>>>>>>> 4.1 via:
>>>>>>>>>>
>>>>>>>>>> $ ./run_suite.sh basic-suite-4.1/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> OK, then we need to investigate it and understand why add host
>>>>>>>>> fails on 4.1, please provide logs when possible.
>>>>>>>>>
>>>>>>>>> BTW, you can also try running the manual job [1], but you'll need
>>>>>>>>> to provide it with custom yum repo URL with your built artifacts to test
>>>>>>>>> your code.
>>>>>>>>> There is a section on it on the OST readthedocs page.
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 14, 2017 at 12:29 AM, Eyal Edri <eedri at redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I see you run the 'hc' suite, which means hyperconverged. This
>>>>>>>>>>> suite is running an hosted engine on gluster storage, It's a more complex
>>>>>>>>>>> suite than the basic one, and prone to more errors, however it should still
>>>>>>>>>>> work, but if you don't require it specifically, I would recommend running
>>>>>>>>>>> the basic suite, which should be easier to debug and also quicker to run.
>>>>>>>>>>>
>>>>>>>>>>> I'm also adding hc maintainer so she can check, can you share the
>>>>>>>>>>> link to the Jenkins job you're running? Or the log files if you're running
>>>>>>>>>>> locally.
>>>>>>>>>>>
>>>>>>>>>>> On Aug 14, 2017 06:00, "Marc Young" <3vilpenguin at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Actually I spoke too soon, still fails:
>>>>>>>>>>>>
>>>>>>>>>>>> + lago ovirt runtest
>>>>>>>>>>>> /home/myoung/repos/github/ovirt-system-tests/vagrant/test-scenarios/002_bootstrap.py
>>>>>>>>>>>> @ Run test: 002_bootstrap.py:
>>>>>>>>>>>> nose.config: INFO: Ignoring files matching ['^\\.', '^_',
>>>>>>>>>>>> '^setup\\.py$']
>>>>>>>>>>>> # print_api_ver:
>>>>>>>>>>>> # print_api_ver: Success (in 0:00:00)
>>>>>>>>>>>> # add_dc:
>>>>>>>>>>>> # add_dc: Success (in 0:00:43)
>>>>>>>>>>>> # add_cluster:
>>>>>>>>>>>> # add_cluster: Success (in 0:00:03)
>>>>>>>>>>>> # add_hosts:
>>>>>>>>>>>> dd
>>>>>>>>>>>> * Collect artifacts:
>>>>>>>>>>>> * Collect artifacts: Success (in 0:01:14)
>>>>>>>>>>>> # add_hosts: Success (in 0:16:36)
>>>>>>>>>>>> # Results located at
>>>>>>>>>>>> /home/myoung/repos/github/ovirt-system-tests/deployment-vagrant/default/002_bootstrap.py.junit.xml
>>>>>>>>>>>> @ Run test: 002_bootstrap.py: Success (in 0:17:26)
>>>>>>>>>>>> Error occured, aborting
>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line
>>>>>>>>>>>> 360, in do_run
>>>>>>>>>>>> self.cli_plugins[args.ovirtverb].do_run(args)
>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py",
>>>>>>>>>>>> line 184, in do_run
>>>>>>>>>>>> self._do_run(**vars(args))
>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line
>>>>>>>>>>>> 501, in wrapper
>>>>>>>>>>>> return func(*args, **kwargs)
>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line
>>>>>>>>>>>> 512, in wrapper
>>>>>>>>>>>> return func(*args, prefix=prefix, **kwargs)
>>>>>>>>>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line
>>>>>>>>>>>> 99, in do_ovirt_runtest
>>>>>>>>>>>> raise RuntimeError('Some tests failed')
>>>>>>>>>>>> RuntimeError: Some tests failed
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Aug 13, 2017 at 9:47 PM, Marc Young
>>>>>>>>>>>> <3vilpenguin at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Edit: reply-all
>>>>>>>>>>>>>
>>>>>>>>>>>>> It does, every time.
>>>>>>>>>>>>> I got it to pass by using the last passing revision from
>>>>>>>>>>>>> Jenkins (hash 98ae6d0b452d098f2703a197deb082a091bba837 ), noted from
>>>>>>>>>>>>> http://jenkins.ovirt.org/job/system-tests_hc-basic-suite-master/15/consoleFull
>>>>>>>>>>>>>
>>>>>>>>>>>>> Not sure if it's a true race condition, that build in Jenkins
>>>>>>>>>>>>> has failed consistently since #15
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Aug 13, 2017 at 2:03 AM, Eyal Edri <eedri at redhat.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 11, 2017 at 9:34 PM, Marc Young
>>>>>>>>>>>>>> <3vilpenguin at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What's the fix for this for those of us using
>>>>>>>>>>>>>>> ovirt-system-tests?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Im trying to adapt some of the code for testing a third party
>>>>>>>>>>>>>>> tool, but master is still failing on 002_bootstrap
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It fails consistently?
>>>>>>>>>>>>>> AFAIK this is a race condition that happens maybe once a week,
>>>>>>>>>>>>>> can you share your logs? is it also failing on add_secondary_storage?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Aug 1, 2017 at 9:20 AM, Benny Zlotnik
>>>>>>>>>>>>>>> <bzlotnik at redhat.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm not sure it's related since the LSM test runs much
>>>>>>>>>>>>>>>> later, in 004
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Aug 1, 2017 at 3:33 PM, Eyal Edri <eedri at redhat.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Adding Allon & Benny.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is it possible to verify this is related to the LSM issue
>>>>>>>>>>>>>>>>> we've been handling in [1]?
>>>>>>>>>>>>>>>>> If this is the case, we agreed to disable the test next
>>>>>>>>>>>>>>>>> time it fails, as the current workarounds with sleep isn't enough.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you confirm this is the case, and so we'll have to
>>>>>>>>>>>>>>>>> disable this test until one of the RFEs described in [1] is merged?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1] https://gerrit.ovirt.org/#/c/78613/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Aug 1, 2017 at 2:45 PM, Barak Korren
>>>>>>>>>>>>>>>>> <bkorren at redhat.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 1 August 2017 at 14:39, Nir Soffer <nsoffer at redhat.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > On Tue, Aug 1, 2017 at 2:34 PM Barak Korren
>>>>>>>>>>>>>>>>>> > <bkorren at redhat.com> wrote:
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> Test failed: [
>>>>>>>>>>>>>>>>>> >> 002_bootstrap.add_secondary_storage_domains ]
>>>>>>>>>>>>>>>>>> >>
>>>>>>>>>>>>>>>>>> >> Link to suspected patches:
>>>>>>>>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/79974
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>> > This patch adds missing log when resizing an online disk
>>>>>>>>>>>>>>>>>> > - why do you think
>>>>>>>>>>>>>>>>>> > it is related to the failure?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Because it is the only patch participating in the test.
>>>>>>>>>>>>>>>>>> (The test was equivalent to running the manual job with
>>>>>>>>>>>>>>>>>> just this patch)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Then again this may also one of the usual SD testing race
>>>>>>>>>>>>>>>>>> conditions.
>>>>>>>>>>>>>>>>>> Since the code in the patch seemed to be related to
>>>>>>>>>>>>>>>>>> storage I didn't
>>>>>>>>>>>>>>>>>> want to just assume that.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Barak Korren
>>>>>>>>>>>>>>>>>> RHV DevOps team , RHCE, RHCi
>>>>>>>>>>>>>>>>>> Red Hat EMEA
>>>>>>>>>>>>>>>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> Devel mailing list
>>>>>>>>>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Eyal edri
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ASSOCIATE MANAGER
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> RHV DevOps
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Red Hat EMEA
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> TRIED. TESTED. TRUSTED.
>>>>>>>>>>>>>>>>> phone: +972-9-7692018
>>>>>>>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Devel mailing list
>>>>>>>>>>>>>>>> Devel at ovirt.org
>>>>>>>>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Eyal edri
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ASSOCIATE MANAGER
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> RHV DevOps
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Red Hat EMEA
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TRIED. TESTED. TRUSTED.
>>>>>>>>>>>>>> phone: +972-9-7692018
>>>>>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Eyal edri
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ASSOCIATE MANAGER
>>>>>>>>>
>>>>>>>>> RHV DevOps
>>>>>>>>>
>>>>>>>>> EMEA VIRTUALIZATION R&D
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Red Hat EMEA
>>>>>>>>>
>>>>>>>>> TRIED. TESTED. TRUSTED.
>>>>>>>>> phone: +972-9-7692018
>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Devel mailing list
>>>>>>>> Devel at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Eyal edri
>>>
>>>
>>> ASSOCIATE MANAGER
>>>
>>> RHV DevOps
>>>
>>> EMEA VIRTUALIZATION R&D
>>>
>>>
>>> Red Hat EMEA
>>>
>>> TRIED. TESTED. TRUSTED.
>>> phone: +972-9-7692018
>>> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>>
>>
>
>
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
More information about the Devel
mailing list