Re: OST fails in 002_bootstrap_pytest.py - setup_storage.sh

On 3/23/20 2:53 PM, Nir Soffer wrote: > On Mon, Mar 23, 2020 at 3:26 PM Marcin Sobczyk <msobczyk(a)redhat.com&gt; > wrote: >> >> >> On 3/23/20 2:17 PM, Nir Soffer wrote: >>> On Mon, Mar 23, 2020 at 1:25 PM Marcin Sobczyk >>> <msobczyk(a)redhat.com&gt; wrote: >>>> >>>> On 3/21/20 1:18 AM, Nir Soffer wrote: >>>> >>>> On Fri, Mar 20, 2020 at 9:35 PM Nir Soffer <nsoffer(a)redhat.com&gt; >>>> wrote: >>>>> Looks like infrastructure issue setting up storage on engine host. >>>>> >>>>> Here are 2 failing builds with unrelated changes: >>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6677/ >>>>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6678/ >>>> Rebuilding still fails in setup_storage: >>>> >>>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6679/testReport/ >>>> >>>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6680/testReport/ >>>> >>>> >>>>> Is this a known issue? >>>>> >>>>> Error Message >>>>> >>>>> AssertionError: setup_storage.sh failed. Exit code is 1 assert 1 >>>>> == 0   -1   +0 >>>>> >>>>> Stacktrace >>>>> >>>>> prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7f6fd2b998d0> >>>>> >>>>>       @pytest.mark.run(order=14) >>>>>       def test_configure_storage(prefix): >>>>>           engine = prefix.virt_env.engine_vm() >>>>>           result = engine.ssh( >>>>>               [ >>>>>                   '/tmp/setup_storage.sh', >>>>>               ], >>>>>           ) >>>>>>         assert result.code == 0, 'setup_storage.sh failed. Exit >>>>>> code is %s' % result.code >>>>> E       AssertionError: setup_storage.sh failed. Exit code is 1 >>>>> E       assert 1 == 0 >>>>> E         -1 >>>>> E         +0 >>>>> >>>>> >>>>> The pytest traceback is nice, but in this case it is does not >>>>> show any useful info. >>>>> >>>>> Since we run a script using ssh, the error message should include >>>>> the process stdout and stderr >>>>> which probably can explain the failure. >>>> I posted https://gerrit.ovirt.org/#/c/107830/ to improve logging >>>> during storage setup. >>>> Unfortunately AFAICS it didn't fail, so I guess we'll have to >>>> merge it and wait for a failed job to get some helpful logs. >>> Thanks. >>> >>> It still fails for me with current code: >>> https://jenkins.ovirt.org/job/ovirt-system-tests_manual/6689/testReport/ >>> >>> >>> Same when using current vdsm master. >> Updated the patch according to your suggestions and currently trying >> out >> OST for the 4th time - >> all previous runs succeeded. I guess I'm out of luck :) > It succeeds on your local OST setup but fail on Jenkins? No, I mean jenkins - both check-patch runs didn't fail on this script. I also tried running OST manually twice and same thing happened. Anyway - the patch has been merged now so if any failure occurs in CQ we should know what's going on.

> >>>>> Also I wonder why this code is called as a test >>>>> (test_configure_storage). This looks like setup >>>>> step so it should run as a fixture. >>>> That's true, but the pytest porting effort was about providing a >>>> bare minimum to move away from nose. >>>> Organizing the tests into proper setup/fixtures is a huge task and >>>> will be probably implemented >>>> incrementally in the nearest future. >>> Understood >>>