failed when running the CI for the patch - https://gerrit.ovirt.org/#/c/93028/ link - http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/24344/ On Sun, Jul 15, 2018 at 4:36 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > On Sun, Jul 15, 2018 at 3:43 PM Eyal Shenitzky <eshenitz(a)redhat.com&gt; > wrote: > >> Did someone encounter this failure? >> >> *12:05:19* 0.00s teardown tests/pywatch_test.py::TestPyWatch::test_kill_grandkids*12:05:19* =================================== FAILURES ===================================*12:05:19* ______________________ TestPyWatch.test_timeout_backtrace ______________________*12:05:19* *12:05:19* self = <pywatch_test.TestPyWatch object at 0x7f0f730219d0>*12:05:19* *12:05:19* @pytest.mark.xfail(on_fedora(), reason="py-bt is broken on Fedora 27")*12:05:19* def test_timeout_backtrace(self):*12:05:19* script = '''*12:05:19* import time*12:05:19* *12:05:19* def outer():*12:05:19* inner()*12:05:19* *12:05:19* def inner():*12:05:19* time.sleep(10)*12:05:19* *12:05:19* outer()*12:05:19* '''*12:05:19* rc, out, err = exec_cmd(['./py-watch', '0.1', 'python', '-c', script])*12:05:19* > assert b'in inner ()' in out*12:05:19* E AssertionError: assert 'in inner ()' in '=============================================================\n= Watched process timed out ... Terminating watched process =\n=============================================================\n'*12:05:19* *12:05:19* pywatch_test.py:68: AssertionError*12:05:19* ------------------------------ Captured log call -------------------------------*12:05:19* cmdutils.py 151 DEBUG ./py-watch 0.1 python -c '*12:05:19* import time*12:05:19* *12:05:19* def outer():*12:05:19* inner()*12:05:19* *12:05:19* def inner():*12:05:19* time.sleep(10)*12:05:19* *12:05:19* outer()*12:05:19* ' (cwd None)*12:05:19* cmdutils.py 159 DEBUG FAILED: <err> = 'Missing separate debuginfo for /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.tox/lib-py27/bin/python2.7\nTry: yum --enablerepo=\'*debug*\' install /usr/lib/debug/.build-id/db/43afedf61f35fbdf896c13894b501cb8bf1dad.debug\nUndefined command: "py-bt". Try "help".\n'; <rc> = 143 >> >> > This used to work on el7. Does it fail in a consistent way now? > > Moving the discussion to devel, this is not storage specific. > > Nir > >> -- Regards, Eyal Shenitzky

On Sun, Jul 15, 2018 at 6:51 PM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > May I repeat Nir's question: does it fail consistently? > And are you rebased on master? > > Undefined command: "py-bt" > > > Is a known xfail for Fedora > > > On Sun, Jul 15, 2018, 17:49 Eyal Shenitzky <eshenitz(a)redhat.com&gt; wrote: > >> failed when running the CI for the patch - >> https://gerrit.ovirt.org/#/c/93028/ >> >> link - >> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/24344/ >> >> According to the build log: *11:58:34* make[1]: Leaving directory `/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'*11:58:34* + debuginfo-install -y python*11:58:39* Could not find debuginfo for main pkg: python-2.7.5-69.el7_5.x86_64*11:58:41* Could not find debuginfo pkg for dependency package python-libs-2.7.5-69.el7_5.x86_64*11:58:47* *11:58:47* ================================================================================*11:58:47* Package Arch Version Repository Size*11:58:47* ================================================================================*11:58:47* Installing:*11:58:47* glibc-debuginfo x86_64 2.17-222.el7 centos-debuginfo 9.5 M*11:58:47* yum-plugin-auto-update-debug-info noarch 1.1.31-45.el7 centos-base-el7 27 k*11:58:47* Installing for dependencies:*11:58:47* glibc-debuginfo-common x86_64 2.17-222.el7 centos-debuginfo 9.6 M*11:58:47* *11:58:47* Transaction Summary*11:58:47* ================================================================================*11:58:47* Install 2 Packages (+1 Dependent package) We could not find python debuginfo package, which explains why py-bt was missing. The build should fail in this case, but "yum install -y" does not fail when a package is missing. Barak, can you suggest a way to fail the build if a package is missing?

> On Sun, Jul 15, 2018 at 4:36 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >> >>> >>> On Sun, Jul 15, 2018 at 3:43 PM Eyal Shenitzky <eshenitz(a)redhat.com&gt; >>> wrote: >>> >>>> Did someone encounter this failure? >>>> >>>> *12:05:19* 0.00s teardown tests/pywatch_test.py::TestPyWatch::test_kill_grandkids*12:05:19* =================================== FAILURES ===================================*12:05:19* ______________________ TestPyWatch.test_timeout_backtrace ______________________*12:05:19* *12:05:19* self = <pywatch_test.TestPyWatch object at 0x7f0f730219d0>*12:05:19* *12:05:19* @pytest.mark.xfail(on_fedora(), reason="py-bt is broken on Fedora 27")*12:05:19* def test_timeout_backtrace(self):*12:05:19* script = '''*12:05:19* import time*12:05:19* *12:05:19* def outer():*12:05:19* inner()*12:05:19* *12:05:19* def inner():*12:05:19* time.sleep(10)*12:05:19* *12:05:19* outer()*12:05:19* '''*12:05:19* rc, out, err = exec_cmd(['./py-watch', '0.1', 'python', '-c', script])*12:05:19* > assert b'in inner ()' in out*12:05:19* E AssertionError: assert 'in inner ()' in '=============================================================\n= Watched process timed out ... Terminating watched process =\n=============================================================\n'*12:05:19* *12:05:19* pywatch_test.py:68: AssertionError*12:05:19* ------------------------------ Captured log call -------------------------------*12:05:19* cmdutils.py 151 DEBUG ./py-watch 0.1 python -c '*12:05:19* import time*12:05:19* *12:05:19* def outer():*12:05:19* inner()*12:05:19* *12:05:19* def inner():*12:05:19* time.sleep(10)*12:05:19* *12:05:19* outer()*12:05:19* ' (cwd None)*12:05:19* cmdutils.py 159 DEBUG FAILED: <err> = 'Missing separate debuginfo for /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.tox/lib-py27/bin/python2.7\nTry: yum --enablerepo=\'*debug*\' install /usr/lib/debug/.build-id/db/43afedf61f35fbdf896c13894b501cb8bf1dad.debug\nUndefined command: "py-bt". Try "help".\n'; <rc> = 143 >>>> >>>> >>> This used to work on el7. Does it fail in a consistent way now? >>> >>> Moving the discussion to devel, this is not storage specific. >>> >>> Nir >>> >>>> >> >> >> -- >> Regards, >> Eyal Shenitzky >> >

On 15 July 2018 at 20:11, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > On Sun, Jul 15, 2018 at 6:51 PM Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > >> May I repeat Nir's question: does it fail consistently? >> And are you rebased on master? >> >> Undefined command: "py-bt" >> >> >> Is a known xfail for Fedora >> >> >> On Sun, Jul 15, 2018, 17:49 Eyal Shenitzky <eshenitz(a)redhat.com&gt; wrote: >> >>> failed when running the CI for the patch - >>> https://gerrit.ovirt.org/#/c/93028/ >>> >>> link - >>> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/24344/ >>> >>> > According to the build log: > > *11:58:34* make[1]: Leaving directory `/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'*11:58:34* + debuginfo-install -y python*11:58:39* Could not find debuginfo for main pkg: python-2.7.5-69.el7_5.x86_64*11:58:41* Could not find debuginfo pkg for dependency package python-libs-2.7.5-69.el7_5.x86_64*11:58:47* *11:58:47* ================================================================================*11:58:47* Package Arch Version Repository Size*11:58:47* ================================================================================*11:58:47* Installing:*11:58:47* glibc-debuginfo x86_64 2.17-222.el7 centos-debuginfo 9.5 M*11:58:47* yum-plugin-auto-update-debug-info noarch 1.1.31-45.el7 centos-base-el7 27 k*11:58:47* Installing for dependencies:*11:58:47* glibc-debuginfo-common x86_64 2.17-222.el7 centos-debuginfo 9.6 M*11:58:47* *11:58:47* Transaction Summary*11:58:47* ================================================================================*11:58:47* Install 2 Packages (+1 Dependent package) > > > We could not find python debuginfo package, which explains why py-bt > was missing. > > The build should fail in this case, but "yum install -y" does not fail > when a package > is missing. > > Barak, can you suggest a way to fail the build if a package is missing? > I suppose adding something like the following to the script would do the trick: rpm -q $PACKAGE || exit 1

> > >> On Sun, Jul 15, 2018 at 4:36 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>> >>>> >>>> On Sun, Jul 15, 2018 at 3:43 PM Eyal Shenitzky <eshenitz(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Did someone encounter this failure? >>>>> >>>>> *12:05:19* 0.00s teardown tests/pywatch_test.py::TestPyWatch::test_kill_grandkids*12:05:19* =================================== FAILURES ===================================*12:05:19* ______________________ TestPyWatch.test_timeout_backtrace ______________________*12:05:19* *12:05:19* self = <pywatch_test.TestPyWatch object at 0x7f0f730219d0>*12:05:19* *12:05:19* @pytest.mark.xfail(on_fedora(), reason="py-bt is broken on Fedora 27")*12:05:19* def test_timeout_backtrace(self):*12:05:19* script = '''*12:05:19* import time*12:05:19* *12:05:19* def outer():*12:05:19* inner()*12:05:19* *12:05:19* def inner():*12:05:19* time.sleep(10)*12:05:19* *12:05:19* outer()*12:05:19* '''*12:05:19* rc, out, err = exec_cmd(['./py-watch', '0.1', 'python', '-c', script])*12:05:19* > assert b'in inner ()' in out*12:05:19* E AssertionError: assert 'in inner ()' in '=============================================================\n= Watched process timed out ... Terminating watched process =\n=============================================================\n'*12:05:19* *12:05:19* pywatch_test.py:68: AssertionError*12:05:19* ------------------------------ Captured log call -------------------------------*12:05:19* cmdutils.py 151 DEBUG ./py-watch 0.1 python -c '*12:05:19* import time*12:05:19* *12:05:19* def outer():*12:05:19* inner()*12:05:19* *12:05:19* def inner():*12:05:19* time.sleep(10)*12:05:19* *12:05:19* outer()*12:05:19* ' (cwd None)*12:05:19* cmdutils.py 159 DEBUG FAILED: <err> = 'Missing separate debuginfo for /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.tox/lib-py27/bin/python2.7\nTry: yum --enablerepo=\'*debug*\' install /usr/lib/debug/.build-id/db/43afedf61f35fbdf896c13894b501cb8bf1dad.debug\nUndefined command: "py-bt". Try "help".\n'; <rc> = 143 >>>>> >>>>> >>>> This used to work on el7. Does it fail in a consistent way now? >>>> >>>> Moving the discussion to devel, this is not storage specific. >>>> >>>> Nir >>>> >>>>> >>> >>> >>> -- >>> Regards, >>> Eyal Shenitzky >>> >> -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

On Mon, Jul 16, 2018 at 8:59 AM Barak Korren <bkorren(a)redhat.com&gt; wrote: > On 15 July 2018 at 20:11, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > >> On Sun, Jul 15, 2018 at 6:51 PM Dan Kenigsberg <danken(a)redhat.com&gt; >> wrote: >> >>> May I repeat Nir's question: does it fail consistently? >>> And are you rebased on master? >>> >>> Undefined command: "py-bt" >>> >>> >>> Is a known xfail for Fedora >>> >>> >>> On Sun, Jul 15, 2018, 17:49 Eyal Shenitzky <eshenitz(a)redhat.com&gt; wrote: >>> >>>> failed when running the CI for the patch - >>>> https://gerrit.ovirt.org/#/c/93028/ >>>> >>>> link - >>>> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/24344/ >>>> >>>> >> According to the build log: >> >> *11:58:34* make[1]: Leaving directory `/home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm'*11:58:34* + debuginfo-install -y python*11:58:39* Could not find debuginfo for main pkg: python-2.7.5-69.el7_5.x86_64*11:58:41* Could not find debuginfo pkg for dependency package python-libs-2.7.5-69.el7_5.x86_64*11:58:47* *11:58:47* ================================================================================*11:58:47* Package Arch Version Repository Size*11:58:47* ================================================================================*11:58:47* Installing:*11:58:47* glibc-debuginfo x86_64 2.17-222.el7 centos-debuginfo 9.5 M*11:58:47* yum-plugin-auto-update-debug-info noarch 1.1.31-45.el7 centos-base-el7 27 k*11:58:47* Installing for dependencies:*11:58:47* glibc-debuginfo-common x86_64 2.17-222.el7 centos-debuginfo 9.6 M*11:58:47* *11:58:47* Transaction Summary*11:58:47* ================================================================================*11:58:47* Install 2 Packages (+1 Dependent package) >> >> >> We could not find python debuginfo package, which explains why py-bt >> was missing. >> >> The build should fail in this case, but "yum install -y" does not fail >> when a package >> is missing. >> >> Barak, can you suggest a way to fail the build if a package is missing? >> > > I suppose adding something like the following to the script would do the > trick: > > rpm -q $PACKAGE || exit 1 > Right, but we don't want to abort the build because python-debuginfo is missing. This is an issue only for pywatch_test.py, so we need to skip the relevant tests when the package is not installed. We can use rpm -q to detect this situation. This is also an issue for the build, because we would not be able to provide a backtrace when the build get stuck, but this practically never happens, so there is no reason to fail the build. The real issue is why python-debuginfo is not available? is this a repo issue? issue with the way we try to install it? > > > >> >> >>> On Sun, Jul 15, 2018 at 4:36 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>>> >>>>> >>>>> On Sun, Jul 15, 2018 at 3:43 PM Eyal Shenitzky <eshenitz(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> Did someone encounter this failure? >>>>>> >>>>>> *12:05:19* 0.00s teardown tests/pywatch_test.py::TestPyWatch::test_kill_grandkids*12:05:19* =================================== FAILURES ===================================*12:05:19* ______________________ TestPyWatch.test_timeout_backtrace ______________________*12:05:19* *12:05:19* self = <pywatch_test.TestPyWatch object at 0x7f0f730219d0>*12:05:19* *12:05:19* @pytest.mark.xfail(on_fedora(), reason="py-bt is broken on Fedora 27")*12:05:19* def test_timeout_backtrace(self):*12:05:19* script = '''*12:05:19* import time*12:05:19* *12:05:19* def outer():*12:05:19* inner()*12:05:19* *12:05:19* def inner():*12:05:19* time.sleep(10)*12:05:19* *12:05:19* outer()*12:05:19* '''*12:05:19* rc, out, err = exec_cmd(['./py-watch', '0.1', 'python', '-c', script])*12:05:19* > assert b'in inner ()' in out*12:05:19* E AssertionError: assert 'in inner ()' in '=============================================================\n= Watched process timed out ... Terminating watched process =\n=============================================================\n'*12:05:19* *12:05:19* pywatch_test.py:68: AssertionError*12:05:19* ------------------------------ Captured log call -------------------------------*12:05:19* cmdutils.py 151 DEBUG ./py-watch 0.1 python -c '*12:05:19* import time*12:05:19* *12:05:19* def outer():*12:05:19* inner()*12:05:19* *12:05:19* def inner():*12:05:19* time.sleep(10)*12:05:19* *12:05:19* outer()*12:05:19* ' (cwd None)*12:05:19* cmdutils.py 159 DEBUG FAILED: <err> = 'Missing separate debuginfo for /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64/vdsm/.tox/lib-py27/bin/python2.7\nTry: yum --enablerepo=\'*debug*\' install /usr/lib/debug/.build-id/db/43afedf61f35fbdf896c13894b501cb8bf1dad.debug\nUndefined command: "py-bt". Try "help".\n'; <rc> = 143 >>>>>> >>>>>> >>>>> This used to work on el7. Does it fail in a consistent way now? >>>>> >>>>> Moving the discussion to devel, this is not storage specific. >>>>> >>>>> Nir >>>>> >>>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Eyal Shenitzky >>>> >>> > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >

On 07/18/2018 12:11 PM, Nir Soffer wrote: On Wed, Jul 18, 2018 at 1:00 PM Francesco Romani <fromani(a)redhat.com&gt; wrote: > On 07/15/2018 05:50 PM, Dan Kenigsberg wrote: > > May I repeat Nir's question: does it fail consistently? > And are you rebased on master? > > Undefined command: "py-bt" > > Is a known xfail for Fedora > > > I'd just like to add that I'm facing this error pretty regularly in the > past couple of days. > > I can provide examples if needed but it seems to me we figured it out. > We can disable the relevant pywatch tests in ovirt CI until this is resolved. Done: https://gerrit.ovirt.org/#/c/93108/

-- Francesco Romani Senior SW Eng., Virtualization R&D Red Hat IRC: fromani github: @fromanirh