[ovirt-devel] Re: [urgent] vdsm broken since Friday (failing CQ)

We have a passing build on master waiting for 4.3 On Thu, Jun 13, 2019 at 9:01 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > Thanks Shirly and Martin. > I see the patch was merged so monitoring and will update > > On Wed, Jun 12, 2019 at 4:09 PM Martin Perina <mperina(a)redhat.com&gt; wrote: > >> >> >> On Wed, Jun 12, 2019 at 11:53 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >> >>> latest tests are still failing on log collector >>> >> >> I had a typo in my patch, here is the fix: >> https://gerrit.ovirt.org/#/c/100784/ >> Currently trying to verify using manual OST to make sure latest VDSM is >> used (and not latest tested as in check-patch executed from the fix) ... >> >> >>> On Wed, Jun 12, 2019 at 10:18 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>> >>>> thanks Martin. >>>> we are failing on missing packages so I have a patch to fix it. >>>> I will update once we have a vdsm build >>>> >>>> On Tue, Jun 11, 2019 at 10:08 PM Martin Perina <mperina(a)redhat.com&gt; >>>> wrote: >>>> >>>>> >>>>> >>>>> On Tue, Jun 11, 2019 at 7:41 PM Martin Perina <mperina(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Tue, Jun 11, 2019 at 6:53 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>> >>>>>>> Galit's patch should have solved it. >>>>>>> Marcin, are you still failing? >>>>>>> >>>>>> >>>>>> I've just rebased https://gerrit.ovirt.org/100716 on top of Galit's >>>>>> change so we should know within an hour or so ... >>>>>> >>>>> >>>>> CI finished successfully, please review so we can merge ... >>>>> >>>>> >>>>>> >>>>>>> >>>>>>> On Tue, Jun 11, 2019 at 2:40 PM Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jun 11, 2019 at 4:35 PM Marcin Sobczyk < >>>>>>>> msobczyk(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> basic suite fails for the patch with the following log: >>>>>>>>> >>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... - STDERR >>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... + yum -y install ovirt-host >>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Error: Package: 32:bind-libs-9.9.4-74.el7_6.1.x86_64 (alocalsync) >>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Requires: bind-license = 32:9.9.4-74.el7_6.1 >>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Installed: 32:bind-license-9.9.4-73.el7_6.noarch (installed) >>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... bind-license = 32:9.9.4-73.el7_6 >>>>>>>>> >>>>>>>>> >>>>>>>> I think https://gerrit.ovirt.org/#/c/100691/ which was merged 2 >>>>>>>> hours ago should address this issue. >>>>>>>> Maybe other reposync files should be updated as well? >>>>>>>> >>>>>>>> >>>>>>>>> Seems unrelated and I'm having the same issue locally on my >>>>>>>>> server. >>>>>>>>> Dafna, do you have some insight regarding this dependency error? >>>>>>>>> >>>>>>>>> Thanks, Marcin >>>>>>>>> On 6/11/19 1:15 PM, Martin Perina wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 11, 2019 at 11:58 AM Milan Zamazal < >>>>>>>>> mzamazal(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>>> Dafna Ron <dron(a)redhat.com&gt; writes: >>>>>>>>>> >>>>>>>>>> > Hi, >>>>>>>>>> > >>>>>>>>>> > Please note vdsm has been broken since Fri the 7th >>>>>>>>>> > >>>>>>>>>> > to summarize again, vdsm has a patch to remove sos plugin >>>>>>>>>> which is what >>>>>>>>>> > metrics is using in its ost tests >>>>>>>>>> > due to that, vdsm is failing the metrics tests and in order to >>>>>>>>>> solve it we >>>>>>>>>> > need to make a choice: >>>>>>>>>> > 1. fix the metrics tests to not use sos >>>>>>>>>> > 2. disable the metrics tests >>>>>>>>>> > 3. revert the sos patch until a decision is made on ^^ >>>>>>>>>> >>>>>>>>>> #3 is not an option, it would make Vdsm uninstallable on newer >>>>>>>>>> RHEL >>>>>>>>>> versions. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I've posted a patch https://gerrit.ovirt.org/100716 which is >>>>>>>>> trying to install vdsm sos plugin if it's not installed either by vdsm nor >>>>>>>>> sos. >>>>>>>>> Currenlty waiting for CI, if run is successfull, I will extend >>>>>>>>> the patch also for 4.3 basic suite. >>>>>>>>> >>>>>>>>> >>>>>>>>>> > Thanks, >>>>>>>>>> > Dafna >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > ---------- Forwarded message --------- >>>>>>>>>> > From: Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>>> > Date: Mon, Jun 10, 2019 at 1:30 PM >>>>>>>>>> > Subject: Re: [ OST Failure Report ] [ oVirt Master (vdsm) ] [ >>>>>>>>>> 07-06-2019 ] >>>>>>>>>> > [ 003_00_metrics_bootstrap.metrics_and_log_collector ] >>>>>>>>>> > To: Martin Perina <mperina(a)redhat.com&gt;, Milan Zamazal < >>>>>>>>>> mzamazal(a)redhat.com&gt;, >>>>>>>>>> > Shirly Radco <sradco(a)redhat.com&gt; >>>>>>>>>> > Cc: devel <devel(a)ovirt.org&gt;, infra <infra(a)ovirt.org&gt; >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > Shirly? any update on this? >>>>>>>>>> > >>>>>>>>>> > On Fri, Jun 7, 2019 at 11:54 AM Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> > >>>>>>>>>> >> Hi, >>>>>>>>>> >> >>>>>>>>>> >> We have a failure in vdsm project on master. >>>>>>>>>> >> >>>>>>>>>> >> The issue is change: >>>>>>>>>> >> https://gerrit.ovirt.org/#/c/100576/ - Remove SOS VDSM plugin >>>>>>>>>> >> >>>>>>>>>> >> which is failing on metrics as metrics is calling >>>>>>>>>> sos-logcollector. >>>>>>>>>> >> >>>>>>>>>> >> The patch cannot be changed as until centos 7.7 when >>>>>>>>>> sos-3.7-3, which >>>>>>>>>> >> contains vdsm plugin will come out. >>>>>>>>>> >> so until then, we are left with no sos plugin, which is >>>>>>>>>> causing the >>>>>>>>>> >> metrics test to fail. >>>>>>>>>> >> >>>>>>>>>> >> Shirly, can you please take a look and see if we can change >>>>>>>>>> the test to >>>>>>>>>> >> not call sos-logcollector? >>>>>>>>>> >> Please note, that we are expecting 4.3 to fail on same issue >>>>>>>>>> very soon. >>>>>>>>>> >> >>>>>>>>>> >> failed job can be found here: >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14452/ >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> ERROR from test: >>>>>>>>>> >> >>>>>>>>>> >> lago.ssh: DEBUG: Command 8626fe70 on >>>>>>>>>> lago-basic-suite-master-engine errors: >>>>>>>>>> >> ERROR: Failed to get a sosreport from: >>>>>>>>>> lago-basic-suite-master-host-1; Could not parse sosreport output >>>>>>>>>> >> ERROR: Failed to get a sosreport from: >>>>>>>>>> lago-basic-suite-master-host-0; Could not parse sosreport output >>>>>>>>>> >> >>>>>>>>>> >> lago.utils: DEBUG: Error while running thread Thread-3 >>>>>>>>>> >> Traceback (most recent call last): >>>>>>>>>> >> File "/usr/lib/python2.7/site-packages/lago/utils.py", line >>>>>>>>>> 58, in _ret_via_queue >>>>>>>>>> >> queue.put({'return': func()}) >>>>>>>>>> >> File >>>>>>>>>> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_00_metrics_bootstrap.py", >>>>>>>>>> line 97, in run_log_collector >>>>>>>>>> >> 'log collector failed. Exit code is %s' % result.code >>>>>>>>>> >> File "/usr/lib64/python2.7/unittest/case.py", line 462, in >>>>>>>>>> assertTrue >>>>>>>>>> >> raise self.failureException(msg) >>>>>>>>>> >> AssertionError: log collector failed. Exit code is 1 >>>>>>>>>> >> --------------------- >> end captured logging << ---------- >>>>>>>>>> >> >>>>>>>>>> >> Thanks, >>>>>>>>>> >> Dafna >>>>>>>>>> >> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Martin Perina >>>>>>>>> Manager, Software Engineering >>>>>>>>> Red Hat Czech s.r.o. >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SQJDJHHQR52... >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>> oVirt Code of Conduct: >>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>> List Archives: >>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GW3E7ZIE3WI... >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Eyal edri >>>>>>>> >>>>>>>> >>>>>>>> MANAGER >>>>>>>> >>>>>>>> CONTINUOUS PRODUCTIZATION >>>>>>>> >>>>>>>> SYSTEM ENGINEERING >>>>>>>> >>>>>>>> Red Hat <https://www.redhat.com/> >>>>>>>> <https://red.ht/sig> >>>>>>>> phone: +972-9-7692018 >>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel) >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> oVirt Code of Conduct: >>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>> List Archives: >>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3TLL7C4LF76... >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Martin Perina >>>>>> Manager, Software Engineering >>>>>> Red Hat Czech s.r.o. >>>>>> >>>>> >>>>> >>>>> -- >>>>> Martin Perina >>>>> Manager, Software Engineering >>>>> Red Hat Czech s.r.o. >>>>> >>>> >> >> -- >> Martin Perina >> Manager, Software Engineering >> Red Hat Czech s.r.o. >> >