On Tue, 27 Nov 2018, 22:49 Nir Soffer <nsoffer(a)redhat.com <mailto:nsoffer@redhat.com> wrote: vdsm check-merged job takes 1.5 hours, and fails 80% of the time. https://jenkins.ovirt.org/job/vdsm_master_check-merged-el7-x86_64/buildTi... Nobody checks these failures or care about them. We are abusing CI resources for no benefit. Can we remove this job? I fought for it quite a lot, but now is the time for me to declare defeat. Unless someone here volunteers to maintain it, we should put the plug out. Nir

On Wed, Nov 28, 2018 at 11:28 AM Marcin Sobczyk <msobczyk(a)redhat.com&gt; wrote: > How much value does it add comparing to check-patch? > > If we can hold for a while with pulling the plug, I can try to split it > into substages in stdci v2 and see if things stabilize a bit. > I would prefer we first work with stdci v2 in order to move the functional tests there (or at least play with it). Then we can remove it.

failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/95559/13

On Wed, Nov 28, 2018 at 11:30 PM Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > On Wed, Nov 28, 2018 at 12:03 PM Edward Haas <edwardh(a)redhat.com&gt; wrote: > >> >> >> On Wed, Nov 28, 2018 at 11:28 AM Marcin Sobczyk <msobczyk(a)redhat.com&gt; >> wrote: >> >>> How much value does it add comparing to check-patch? >>> >>> If we can hold for a while with pulling the plug, I can try to split it >>> into substages in stdci v2 and see if things stabilize a bit. >>> >> >> I would prefer we first work with stdci v2 in order to move the >> functional tests there (or at least play with it). >> Then we can remove it. >> > > Turns out that this worth with stdci v2 - if check-merged fail, the > change queue > will reject the patch. > > See this mail from infra(a)ovirt.org mailing list: > > Change 95559,13 (vdsm) is probably the reason behind recent system test >> failures in the "ovirt-master" change queue and needs to be fixed. >> >> This change had been removed from the testing queue. Artifacts build >> from this >> change will not be released until it is fixed. >> >> For further details about the change see: >> https://gerrit.ovirt.org/#/c/95559/13 > > > According to Dafna and Barak, change queue require that all jobs pass, so > we cannot > have flaky job in the build. > > I hopefully removed it from stdci v2 here: > https://gerrit.ovirt.org/c/95845/ > > I don't think we should even enable check-merged again. All tests must > run *before* we > merge. When cannot work with a job that will randomly fail after merge. > Here is another failure: A system test invoked by the "ovirt-master" change queue including change 95817,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95559,13 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95559,13 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95817/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95559/13 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11719/ We are going to see more failures, since we merged yesterday several patches after the stdci v2 patch: 8e4df87a5 storage: blocksd_test refactored to use pytest monkeypatching ede08ad53 storage: blocksd_test refactore to use pytest.xfail 46aad2375 storage: blocksd_test convertet to pytest b4f2809c0 storage: Improved SD.create() docstring 30b1423e0 virt: use log.warning(), not log.warn() 224ebf092 ci: Added 'stdciv2' configuration file

Guys, The check-merged job is causing lot of noise and failures in CI and CQ. Can we drop it ASAP? and continue to discuss offline if you want to move that functionality to check-patch as part of V2? Also, if we could drop the V1 jobs that would be great so we'll reduce noise from failures there. On Thu, Nov 29, 2018 at 8:39 AM Barak Korren <bkorren(a)redhat.com <mailto:bkorren@redhat.com>> wrote: On Thu, 29 Nov 2018 at 00:29, Nir Soffer <nsoffer(a)redhat.com <mailto:nsoffer@redhat.com>> wrote: On Wed, Nov 28, 2018 at 11:30 PM Nir Soffer <nsoffer(a)redhat.com <mailto:nsoffer@redhat.com>> wrote: On Wed, Nov 28, 2018 at 12:03 PM Edward Haas <edwardh(a)redhat.com <mailto:edwardh@redhat.com>> wrote: On Wed, Nov 28, 2018 at 11:28 AM Marcin Sobczyk <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com>> wrote: How much value does it add comparing to check-patch? If we can hold for a while with pulling the plug, I can try to split it into substages in stdci v2 and see if things stabilize a bit. I would prefer we first work with stdci v2 in order to move the functional tests there (or at least play with it). Then we can remove it. Turns out that this worth with stdci v2 - if check-merged fail, the change queue will reject the patch. See this mail from infra(a)ovirt.org <mailto:infra@ovirt.org> mailing list: Change 95559,13 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/95559/13 According to Dafna and Barak, change queue require that all jobs pass, so we cannot have flaky job in the build. I hopefully removed it from stdci v2 here: https://gerrit.ovirt.org/c/95845/ I don't think we should even enable check-merged again. All tests must run *before* we merge. When cannot work with a job that will randomly fail after merge. Here is another failure: A system test invoked by the "ovirt-master" change queue including change 95817,2 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 95559,13 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 95559,13 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/95817/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/95559/13 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11719/ We are going to see more failures, since we merged yesterday several patches after the stdci v2 patch: 8e4df87a5 storage: blocksd_test refactored to use pytest monkeypatching ede08ad53 storage: blocksd_test refactore to use pytest.xfail 46aad2375 storage: blocksd_test convertet to pytest b4f2809c0 storage: Improved SD.create() docstring 30b1423e0 virt: use log.warning(), not log.warn() 224ebf092 ci: Added 'stdciv2' configuration file Hold on, since you also have V1 jobs now, patches are being submitted twice into the queue, and there are actually passing when submitted by the v1 jobs like they did before: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11713/exec... So yeah, you should fix check-merged or disable it, but as long as you have the v1 jobs these failures are not causing any real harm, just noise. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com <http://redhat.com> | TRIED. TESTED. TRUSTED. | redhat.com/trusted <http://redhat.com/trusted> _______________________________________________ Devel mailing list -- devel(a)ovirt.org <mailto:devel@ovirt.org> To unsubscribe send an email to devel-leave(a)ovirt.org <mailto:devel-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RVTPRQURXCV... -- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

On Thu, 6 Dec 2018 at 12:10, Marcin Sobczyk <msobczyk(a)redhat.com <mailto:msobczyk@redhat.com>> wrote: Hi Eyal, removal of V1 is on the way: https://gerrit.ovirt.org/#/c/96031/ https://gerrit.ovirt.org/#/c/96027/ https://gerrit.ovirt.org/#/c/95774/ and here's a patch that disables 'check-merged' on master: https://gerrit.ovirt.org/#/c/96032/ Do we want to also disable it on 'ovirt-4.2'? Ether, disable, or make sure it passes.... Marcin On 12/6/18 10:50 AM, Eyal Edri wrote: > Guys, > The check-merged job is causing lot of noise and failures in CI > and CQ. > Can we drop it ASAP? and continue to discuss offline if you want > to move that functionality to check-patch as part of V2? > > Also, if we could drop the V1 jobs that would be great so we'll > reduce noise from failures there. > > On Thu, Nov 29, 2018 at 8:39 AM Barak Korren <bkorren(a)redhat.com > <mailto:bkorren@redhat.com>> wrote: > > > > On Thu, 29 Nov 2018 at 00:29, Nir Soffer <nsoffer(a)redhat.com > <mailto:nsoffer@redhat.com>> wrote: > > On Wed, Nov 28, 2018 at 11:30 PM Nir Soffer > <nsoffer(a)redhat.com <mailto:nsoffer@redhat.com>> wrote: > > On Wed, Nov 28, 2018 at 12:03 PM Edward Haas > <edwardh(a)redhat.com <mailto:edwardh@redhat.com>> wrote: > > > > On Wed, Nov 28, 2018 at 11:28 AM Marcin Sobczyk > <msobczyk(a)redhat.com > <mailto:msobczyk@redhat.com>> wrote: > > How much value does it add comparing to > check-patch? > > If we can hold for a while with pulling the > plug, I can try to split it into substages in > stdci v2 and see if things stabilize a bit. > > > I would prefer we first work with stdci v2 in > order to move the functional tests there (or at > least play with it). > Then we can remove it. > > > Turns out that this worth with stdci v2 - if > check-merged fail, the change queue > will reject the patch. > > See this mail from infra(a)ovirt.org > <mailto:infra@ovirt.org> mailing list: > > Change 95559,13 (vdsm) is probably the reason > behind recent system test > failures in the "ovirt-master" change queue and > needs to be fixed. > > This change had been removed from the testing > queue. Artifacts build from this > change will not be released until it is fixed. > > For further details about the change see: > https://gerrit.ovirt.org/#/c/95559/13 > > > According to Dafna and Barak, change queue require > that all jobs pass, so we cannot > have flaky job in the build. > > I hopefully removed it from stdci v2 here: > https://gerrit.ovirt.org/c/95845/ > > I don't think we should even enable check-merged > again. All tests must run *before* we > merge. When cannot work with a job that will randomly > fail after merge. > > > Here is another failure: > > A system test invoked by the "ovirt-master" change queue > including change > 95817,2 (vdsm) failed. However, this change seems not to > be the root cause for > this failure. Change 95559,13 (vdsm) that this change > depends on or is based > on, was detected as the cause of the testing failures. > > This change had been removed from the testing queue. > Artifacts built from this > change will not be released until either change 95559,13 > (vdsm) is fixed and > this change is updated to refer to or rebased on the > fixed version, or this > change is modified to no longer depend on it. > > For further details about the change see: > https://gerrit.ovirt.org/#/c/95817/2 > > For further details about the change that seems to be the > root cause behind the > testing failures see: > https://gerrit.ovirt.org/#/c/95559/13 > > For failed test results see: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11719/ > > > > We are going to see more failures, since we merged > yesterday several > patches after the stdci v2 patch: > > 8e4df87a5 storage: blocksd_test refactored to use pytest > monkeypatching > ede08ad53 storage: blocksd_test refactore to use pytest.xfail > 46aad2375 storage: blocksd_test convertet to pytest > b4f2809c0 storage: Improved SD.create() docstring > 30b1423e0 virt: use log.warning(), not log.warn() > 224ebf092 ci: Added 'stdciv2' configuration file > > > Hold on, since you also have V1 jobs now, patches are being > submitted twice into the queue, and there are actually > passing when submitted by the v1 jobs like they did before: > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/11713/exec... > > So yeah, you should fix check-merged or disable it, but as > long as you have the v1 jobs these failures are not causing > any real harm, just noise. > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com <http://redhat.com> | TRIED. TESTED. TRUSTED. | > redhat.com/trusted <http://redhat.com/trusted> > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org <mailto:devel@ovirt.org> > To unsubscribe send an email to devel-leave(a)ovirt.org > <mailto:devel-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RVTPRQURXCV... > > > > -- > > Eyal edri > > > MANAGER > > RHV/CNV DevOps > > EMEA VIRTUALIZATION R&D > > > Red Hat EMEA <https://www.redhat.com/> > > <https://red.ht/sig> TRIED. TESTED. TRUSTED. > <https://redhat.com/trusted> > > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com <http://redhat.com> | TRIED. TESTED. TRUSTED. | redhat.com/trusted <http://redhat.com/trusted>