Re: [ovirt-devel] Re: [urgent] vdsm broken since Friday (failing CQ)

On Thu, Jun 13, 2019 at 2:02 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > 4.3 is still failing. > > https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change... > Martin, can you apply the change to 4.3 tests as well? > Arghhh. I looked if master and 4.3 shares the same file when I started to work on the fix and they were the same. But at the same day I posted orignal patch, master and 4.3 were split :-( So hopefully here's the final fix: https://gerrit.ovirt.org/100809 > On Thu, Jun 13, 2019 at 10:33 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> We have a passing build on master >> waiting for 4.3 >> >> On Thu, Jun 13, 2019 at 9:01 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >> >>> Thanks Shirly and Martin. >>> I see the patch was merged so monitoring and will update >>> >>> On Wed, Jun 12, 2019 at 4:09 PM Martin Perina <mperina(a)redhat.com&gt; >>> wrote: >>> >>>> >>>> >>>> On Wed, Jun 12, 2019 at 11:53 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>>> latest tests are still failing on log collector >>>>> >>>> >>>> I had a typo in my patch, here is the fix: >>>> https://gerrit.ovirt.org/#/c/100784/ >>>> Currently trying to verify using manual OST to make sure latest VDSM >>>> is used (and not latest tested as in check-patch executed from the fix) ... >>>> >>>> >>>>> On Wed, Jun 12, 2019 at 10:18 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>> >>>>>> thanks Martin. >>>>>> we are failing on missing packages so I have a patch to fix it. >>>>>> I will update once we have a vdsm build >>>>>> >>>>>> On Tue, Jun 11, 2019 at 10:08 PM Martin Perina <mperina(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Jun 11, 2019 at 7:41 PM Martin Perina <mperina(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jun 11, 2019 at 6:53 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> Galit's patch should have solved it. >>>>>>>>> Marcin, are you still failing? >>>>>>>>> >>>>>>>> >>>>>>>> I've just rebased https://gerrit.ovirt.org/100716 on top of >>>>>>>> Galit's change so we should know within an hour or so ... >>>>>>>> >>>>>>> >>>>>>> CI finished successfully, please review so we can merge ... >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 11, 2019 at 2:40 PM Eyal Edri <eedri(a)redhat.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jun 11, 2019 at 4:35 PM Marcin Sobczyk < >>>>>>>>>> msobczyk(a)redhat.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> basic suite fails for the patch with the following log: >>>>>>>>>>> >>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... - STDERR >>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... + yum -y install ovirt-host >>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Error: Package: 32:bind-libs-9.9.4-74.el7_6.1.x86_64 (alocalsync) >>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Requires: bind-license = 32:9.9.4-74.el7_6.1 >>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Installed: 32:bind-license-9.9.4-73.el7_6.noarch (installed) >>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... bind-license = 32:9.9.4-73.el7_6 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> I think https://gerrit.ovirt.org/#/c/100691/ which was merged 2 >>>>>>>>>> hours ago should address this issue. >>>>>>>>>> Maybe other reposync files should be updated as well? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Seems unrelated and I'm having the same issue locally on my >>>>>>>>>>> server. >>>>>>>>>>> Dafna, do you have some insight regarding this dependency error? >>>>>>>>>>> >>>>>>>>>>> Thanks, Marcin >>>>>>>>>>> On 6/11/19 1:15 PM, Martin Perina wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Jun 11, 2019 at 11:58 AM Milan Zamazal < >>>>>>>>>>> mzamazal(a)redhat.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> Dafna Ron <dron(a)redhat.com&gt; writes: >>>>>>>>>>>> >>>>>>>>>>>> > Hi, >>>>>>>>>>>> > >>>>>>>>>>>> > Please note vdsm has been broken since Fri the 7th >>>>>>>>>>>> > >>>>>>>>>>>> > to summarize again, vdsm has a patch to remove sos plugin >>>>>>>>>>>> which is what >>>>>>>>>>>> > metrics is using in its ost tests >>>>>>>>>>>> > due to that, vdsm is failing the metrics tests and in order >>>>>>>>>>>> to solve it we >>>>>>>>>>>> > need to make a choice: >>>>>>>>>>>> > 1. fix the metrics tests to not use sos >>>>>>>>>>>> > 2. disable the metrics tests >>>>>>>>>>>> > 3. revert the sos patch until a decision is made on ^^ >>>>>>>>>>>> >>>>>>>>>>>> #3 is not an option, it would make Vdsm uninstallable on newer >>>>>>>>>>>> RHEL >>>>>>>>>>>> versions. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I've posted a patch https://gerrit.ovirt.org/100716 which is >>>>>>>>>>> trying to install vdsm sos plugin if it's not installed either by vdsm nor >>>>>>>>>>> sos. >>>>>>>>>>> Currenlty waiting for CI, if run is successfull, I will extend >>>>>>>>>>> the patch also for 4.3 basic suite. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> > Thanks, >>>>>>>>>>>> > Dafna >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > ---------- Forwarded message --------- >>>>>>>>>>>> > From: Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>>>>> > Date: Mon, Jun 10, 2019 at 1:30 PM >>>>>>>>>>>> > Subject: Re: [ OST Failure Report ] [ oVirt Master (vdsm) ] >>>>>>>>>>>> [ 07-06-2019 ] >>>>>>>>>>>> > [ 003_00_metrics_bootstrap.metrics_and_log_collector ] >>>>>>>>>>>> > To: Martin Perina <mperina(a)redhat.com&gt;, Milan Zamazal < >>>>>>>>>>>> mzamazal(a)redhat.com&gt;, >>>>>>>>>>>> > Shirly Radco <sradco(a)redhat.com&gt; >>>>>>>>>>>> > Cc: devel <devel(a)ovirt.org&gt;, infra <infra(a)ovirt.org&gt; >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > Shirly? any update on this? >>>>>>>>>>>> > >>>>>>>>>>>> > On Fri, Jun 7, 2019 at 11:54 AM Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>>>>> wrote: >>>>>>>>>>>> > >>>>>>>>>>>> >> Hi, >>>>>>>>>>>> >> >>>>>>>>>>>> >> We have a failure in vdsm project on master. >>>>>>>>>>>> >> >>>>>>>>>>>> >> The issue is change: >>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/100576/ - Remove SOS VDSM >>>>>>>>>>>> plugin >>>>>>>>>>>> >> >>>>>>>>>>>> >> which is failing on metrics as metrics is calling >>>>>>>>>>>> sos-logcollector. >>>>>>>>>>>> >> >>>>>>>>>>>> >> The patch cannot be changed as until centos 7.7 when >>>>>>>>>>>> sos-3.7-3, which >>>>>>>>>>>> >> contains vdsm plugin will come out. >>>>>>>>>>>> >> so until then, we are left with no sos plugin, which is >>>>>>>>>>>> causing the >>>>>>>>>>>> >> metrics test to fail. >>>>>>>>>>>> >> >>>>>>>>>>>> >> Shirly, can you please take a look and see if we can change >>>>>>>>>>>> the test to >>>>>>>>>>>> >> not call sos-logcollector? >>>>>>>>>>>> >> Please note, that we are expecting 4.3 to fail on same >>>>>>>>>>>> issue very soon. >>>>>>>>>>>> >> >>>>>>>>>>>> >> failed job can be found here: >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14452/ >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> ERROR from test: >>>>>>>>>>>> >> >>>>>>>>>>>> >> lago.ssh: DEBUG: Command 8626fe70 on >>>>>>>>>>>> lago-basic-suite-master-engine errors: >>>>>>>>>>>> >> ERROR: Failed to get a sosreport from: >>>>>>>>>>>> lago-basic-suite-master-host-1; Could not parse sosreport output >>>>>>>>>>>> >> ERROR: Failed to get a sosreport from: >>>>>>>>>>>> lago-basic-suite-master-host-0; Could not parse sosreport output >>>>>>>>>>>> >> >>>>>>>>>>>> >> lago.utils: DEBUG: Error while running thread Thread-3 >>>>>>>>>>>> >> Traceback (most recent call last): >>>>>>>>>>>> >> File "/usr/lib/python2.7/site-packages/lago/utils.py", >>>>>>>>>>>> line 58, in _ret_via_queue >>>>>>>>>>>> >> queue.put({'return': func()}) >>>>>>>>>>>> >> File >>>>>>>>>>>> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_00_metrics_bootstrap.py", >>>>>>>>>>>> line 97, in run_log_collector >>>>>>>>>>>> >> 'log collector failed. Exit code is %s' % result.code >>>>>>>>>>>> >> File "/usr/lib64/python2.7/unittest/case.py", line 462, >>>>>>>>>>>> in assertTrue >>>>>>>>>>>> >> raise self.failureException(msg) >>>>>>>>>>>> >> AssertionError: log collector failed. Exit code is 1 >>>>>>>>>>>> >> --------------------- >> end captured logging << ---------- >>>>>>>>>>>> >> >>>>>>>>>>>> >> Thanks, >>>>>>>>>>>> >> Dafna >>>>>>>>>>>> >> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Martin Perina >>>>>>>>>>> Manager, Software Engineering >>>>>>>>>>> Red Hat Czech s.r.o. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SQJDJHHQR52... >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>> List Archives: >>>>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GW3E7ZIE3WI... >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Eyal edri >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> MANAGER >>>>>>>>>> >>>>>>>>>> CONTINUOUS PRODUCTIZATION >>>>>>>>>> >>>>>>>>>> SYSTEM ENGINEERING >>>>>>>>>> >>>>>>>>>> Red Hat <https://www.redhat.com/> >>>>>>>>>> <https://red.ht/sig> >>>>>>>>>> phone: +972-9-7692018 >>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel) >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>> oVirt Code of Conduct: >>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>> List Archives: >>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3TLL7C4LF76... >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Martin Perina >>>>>>>> Manager, Software Engineering >>>>>>>> Red Hat Czech s.r.o. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Martin Perina >>>>>>> Manager, Software Engineering >>>>>>> Red Hat Czech s.r.o. >>>>>>> >>>>>> >>>> >>>> -- >>>> Martin Perina >>>> Manager, Software Engineering >>>> Red Hat Czech s.r.o. >>>> >>> -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.