Re: [ovirt-devel] Re: [urgent] vdsm broken since Friday (failing CQ)

4.3 is still failing. https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change... Martin, can you apply the change to 4.3 tests as well?

On Thu, Jun 13, 2019 at 10:33 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > We have a passing build on master > waiting for 4.3 > > On Thu, Jun 13, 2019 at 9:01 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> Thanks Shirly and Martin. >> I see the patch was merged so monitoring and will update >> >> On Wed, Jun 12, 2019 at 4:09 PM Martin Perina <mperina(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Wed, Jun 12, 2019 at 11:53 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>> >>>> latest tests are still failing on log collector >>>> >>> >>> I had a typo in my patch, here is the fix: >>> https://gerrit.ovirt.org/#/c/100784/ >>> Currently trying to verify using manual OST to make sure latest VDSM is >>> used (and not latest tested as in check-patch executed from the fix) ... >>> >>> >>>> On Wed, Jun 12, 2019 at 10:18 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>>> thanks Martin. >>>>> we are failing on missing packages so I have a patch to fix it. >>>>> I will update once we have a vdsm build >>>>> >>>>> On Tue, Jun 11, 2019 at 10:08 PM Martin Perina <mperina(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Tue, Jun 11, 2019 at 7:41 PM Martin Perina <mperina(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 11, 2019 at 6:53 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> Galit's patch should have solved it. >>>>>>>> Marcin, are you still failing? >>>>>>>> >>>>>>> >>>>>>> I've just rebased https://gerrit.ovirt.org/100716 on top of >>>>>>> Galit's change so we should know within an hour or so ... >>>>>>> >>>>>> >>>>>> CI finished successfully, please review so we can merge ... >>>>>> >>>>>> >>>>>>> >>>>>>>> >>>>>>>> On Tue, Jun 11, 2019 at 2:40 PM Eyal Edri <eedri(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 11, 2019 at 4:35 PM Marcin Sobczyk < >>>>>>>>> msobczyk(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> basic suite fails for the patch with the following log: >>>>>>>>>> >>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... - STDERR >>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... + yum -y install ovirt-host >>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Error: Package: 32:bind-libs-9.9.4-74.el7_6.1.x86_64 (alocalsync) >>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Requires: bind-license = 32:9.9.4-74.el7_6.1 >>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Installed: 32:bind-license-9.9.4-73.el7_6.noarch (installed) >>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... bind-license = 32:9.9.4-73.el7_6 >>>>>>>>>> >>>>>>>>>> >>>>>>>>> I think https://gerrit.ovirt.org/#/c/100691/ which was merged 2 >>>>>>>>> hours ago should address this issue. >>>>>>>>> Maybe other reposync files should be updated as well? >>>>>>>>> >>>>>>>>> >>>>>>>>>> Seems unrelated and I'm having the same issue locally on my >>>>>>>>>> server. >>>>>>>>>> Dafna, do you have some insight regarding this dependency error? >>>>>>>>>> >>>>>>>>>> Thanks, Marcin >>>>>>>>>> On 6/11/19 1:15 PM, Martin Perina wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jun 11, 2019 at 11:58 AM Milan Zamazal < >>>>>>>>>> mzamazal(a)redhat.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Dafna Ron <dron(a)redhat.com&gt; writes: >>>>>>>>>>> >>>>>>>>>>> > Hi, >>>>>>>>>>> > >>>>>>>>>>> > Please note vdsm has been broken since Fri the 7th >>>>>>>>>>> > >>>>>>>>>>> > to summarize again, vdsm has a patch to remove sos plugin >>>>>>>>>>> which is what >>>>>>>>>>> > metrics is using in its ost tests >>>>>>>>>>> > due to that, vdsm is failing the metrics tests and in order >>>>>>>>>>> to solve it we >>>>>>>>>>> > need to make a choice: >>>>>>>>>>> > 1. fix the metrics tests to not use sos >>>>>>>>>>> > 2. disable the metrics tests >>>>>>>>>>> > 3. revert the sos patch until a decision is made on ^^ >>>>>>>>>>> >>>>>>>>>>> #3 is not an option, it would make Vdsm uninstallable on newer >>>>>>>>>>> RHEL >>>>>>>>>>> versions. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I've posted a patch https://gerrit.ovirt.org/100716 which is >>>>>>>>>> trying to install vdsm sos plugin if it's not installed either by vdsm nor >>>>>>>>>> sos. >>>>>>>>>> Currenlty waiting for CI, if run is successfull, I will extend >>>>>>>>>> the patch also for 4.3 basic suite. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> > Thanks, >>>>>>>>>>> > Dafna >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > ---------- Forwarded message --------- >>>>>>>>>>> > From: Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>>>> > Date: Mon, Jun 10, 2019 at 1:30 PM >>>>>>>>>>> > Subject: Re: [ OST Failure Report ] [ oVirt Master (vdsm) ] [ >>>>>>>>>>> 07-06-2019 ] >>>>>>>>>>> > [ 003_00_metrics_bootstrap.metrics_and_log_collector ] >>>>>>>>>>> > To: Martin Perina <mperina(a)redhat.com&gt;, Milan Zamazal < >>>>>>>>>>> mzamazal(a)redhat.com&gt;, >>>>>>>>>>> > Shirly Radco <sradco(a)redhat.com&gt; >>>>>>>>>>> > Cc: devel <devel(a)ovirt.org&gt;, infra <infra(a)ovirt.org&gt; >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > Shirly? any update on this? >>>>>>>>>>> > >>>>>>>>>>> > On Fri, Jun 7, 2019 at 11:54 AM Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> >> Hi, >>>>>>>>>>> >> >>>>>>>>>>> >> We have a failure in vdsm project on master. >>>>>>>>>>> >> >>>>>>>>>>> >> The issue is change: >>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/100576/ - Remove SOS VDSM >>>>>>>>>>> plugin >>>>>>>>>>> >> >>>>>>>>>>> >> which is failing on metrics as metrics is calling >>>>>>>>>>> sos-logcollector. >>>>>>>>>>> >> >>>>>>>>>>> >> The patch cannot be changed as until centos 7.7 when >>>>>>>>>>> sos-3.7-3, which >>>>>>>>>>> >> contains vdsm plugin will come out. >>>>>>>>>>> >> so until then, we are left with no sos plugin, which is >>>>>>>>>>> causing the >>>>>>>>>>> >> metrics test to fail. >>>>>>>>>>> >> >>>>>>>>>>> >> Shirly, can you please take a look and see if we can change >>>>>>>>>>> the test to >>>>>>>>>>> >> not call sos-logcollector? >>>>>>>>>>> >> Please note, that we are expecting 4.3 to fail on same issue >>>>>>>>>>> very soon. >>>>>>>>>>> >> >>>>>>>>>>> >> failed job can be found here: >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14452/ >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> ERROR from test: >>>>>>>>>>> >> >>>>>>>>>>> >> lago.ssh: DEBUG: Command 8626fe70 on >>>>>>>>>>> lago-basic-suite-master-engine errors: >>>>>>>>>>> >> ERROR: Failed to get a sosreport from: >>>>>>>>>>> lago-basic-suite-master-host-1; Could not parse sosreport output >>>>>>>>>>> >> ERROR: Failed to get a sosreport from: >>>>>>>>>>> lago-basic-suite-master-host-0; Could not parse sosreport output >>>>>>>>>>> >> >>>>>>>>>>> >> lago.utils: DEBUG: Error while running thread Thread-3 >>>>>>>>>>> >> Traceback (most recent call last): >>>>>>>>>>> >> File "/usr/lib/python2.7/site-packages/lago/utils.py", >>>>>>>>>>> line 58, in _ret_via_queue >>>>>>>>>>> >> queue.put({'return': func()}) >>>>>>>>>>> >> File >>>>>>>>>>> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_00_metrics_bootstrap.py", >>>>>>>>>>> line 97, in run_log_collector >>>>>>>>>>> >> 'log collector failed. Exit code is %s' % result.code >>>>>>>>>>> >> File "/usr/lib64/python2.7/unittest/case.py", line 462, in >>>>>>>>>>> assertTrue >>>>>>>>>>> >> raise self.failureException(msg) >>>>>>>>>>> >> AssertionError: log collector failed. Exit code is 1 >>>>>>>>>>> >> --------------------- >> end captured logging << ---------- >>>>>>>>>>> >> >>>>>>>>>>> >> Thanks, >>>>>>>>>>> >> Dafna >>>>>>>>>>> >> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Martin Perina >>>>>>>>>> Manager, Software Engineering >>>>>>>>>> Red Hat Czech s.r.o. >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SQJDJHHQR52... >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GW3E7ZIE3WI... >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Eyal edri >>>>>>>>> >>>>>>>>> >>>>>>>>> MANAGER >>>>>>>>> >>>>>>>>> CONTINUOUS PRODUCTIZATION >>>>>>>>> >>>>>>>>> SYSTEM ENGINEERING >>>>>>>>> >>>>>>>>> Red Hat <https://www.redhat.com/> >>>>>>>>> <https://red.ht/sig> >>>>>>>>> phone: +972-9-7692018 >>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel) >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3TLL7C4LF76... >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Martin Perina >>>>>>> Manager, Software Engineering >>>>>>> Red Hat Czech s.r.o. >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Martin Perina >>>>>> Manager, Software Engineering >>>>>> Red Hat Czech s.r.o. >>>>>> >>>>> >>> >>> -- >>> Martin Perina >>> Manager, Software Engineering >>> Red Hat Czech s.r.o. >>> >>