[ovirt-devel] Re: [urgent] vdsm broken since Friday (failing CQ)

will monitor new change once merged :)

On Thu, Jun 13, 2019 at 1:23 PM Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Thu, Jun 13, 2019 at 2:02 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> 4.3 is still failing. >> >> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change... >> Martin, can you apply the change to 4.3 tests as well? >> > > Arghhh. > > I looked if master and 4.3 shares the same file when I started to work on > the fix and they were the same. > But at the same day I posted orignal patch, master and 4.3 were split :-( > > So hopefully here's the final fix: https://gerrit.ovirt.org/100809 >

> >> On Thu, Jun 13, 2019 at 10:33 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >> >>> We have a passing build on master >>> waiting for 4.3 >>> >>> On Thu, Jun 13, 2019 at 9:01 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>> >>>> Thanks Shirly and Martin. >>>> I see the patch was merged so monitoring and will update >>>> >>>> On Wed, Jun 12, 2019 at 4:09 PM Martin Perina <mperina(a)redhat.com&gt; >>>> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Jun 12, 2019 at 11:53 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>> >>>>>> latest tests are still failing on log collector >>>>>> >>>>> >>>>> I had a typo in my patch, here is the fix: >>>>> https://gerrit.ovirt.org/#/c/100784/ >>>>> Currently trying to verify using manual OST to make sure latest VDSM >>>>> is used (and not latest tested as in check-patch executed from the fix) ... >>>>> >>>>> >>>>>> On Wed, Jun 12, 2019 at 10:18 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>> >>>>>>> thanks Martin. >>>>>>> we are failing on missing packages so I have a patch to fix it. >>>>>>> I will update once we have a vdsm build >>>>>>> >>>>>>> On Tue, Jun 11, 2019 at 10:08 PM Martin Perina <mperina(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Jun 11, 2019 at 7:41 PM Martin Perina <mperina(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Jun 11, 2019 at 6:53 PM Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Galit's patch should have solved it. >>>>>>>>>> Marcin, are you still failing? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I've just rebased https://gerrit.ovirt.org/100716 on top of >>>>>>>>> Galit's change so we should know within an hour or so ... >>>>>>>>> >>>>>>>> >>>>>>>> CI finished successfully, please review so we can merge ... >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jun 11, 2019 at 2:40 PM Eyal Edri <eedri(a)redhat.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Jun 11, 2019 at 4:35 PM Marcin Sobczyk < >>>>>>>>>>> msobczyk(a)redhat.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> basic suite fails for the patch with the following log: >>>>>>>>>>>> >>>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... - STDERR >>>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... + yum -y install ovirt-host >>>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Error: Package: 32:bind-libs-9.9.4-74.el7_6.1.x86_64 (alocalsync) >>>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Requires: bind-license = 32:9.9.4-74.el7_6.1 >>>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... Installed: 32:bind-license-9.9.4-73.el7_6.noarch (installed) >>>>>>>>>>>> <https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... bind-license = 32:9.9.4-73.el7_6 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> I think https://gerrit.ovirt.org/#/c/100691/ which was merged >>>>>>>>>>> 2 hours ago should address this issue. >>>>>>>>>>> Maybe other reposync files should be updated as well? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Seems unrelated and I'm having the same issue locally on my >>>>>>>>>>>> server. >>>>>>>>>>>> Dafna, do you have some insight regarding this dependency >>>>>>>>>>>> error? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, Marcin >>>>>>>>>>>> On 6/11/19 1:15 PM, Martin Perina wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jun 11, 2019 at 11:58 AM Milan Zamazal < >>>>>>>>>>>> mzamazal(a)redhat.com&gt; wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Dafna Ron <dron(a)redhat.com&gt; writes: >>>>>>>>>>>>> >>>>>>>>>>>>> > Hi, >>>>>>>>>>>>> > >>>>>>>>>>>>> > Please note vdsm has been broken since Fri the 7th >>>>>>>>>>>>> > >>>>>>>>>>>>> > to summarize again, vdsm has a patch to remove sos plugin >>>>>>>>>>>>> which is what >>>>>>>>>>>>> > metrics is using in its ost tests >>>>>>>>>>>>> > due to that, vdsm is failing the metrics tests and in order >>>>>>>>>>>>> to solve it we >>>>>>>>>>>>> > need to make a choice: >>>>>>>>>>>>> > 1. fix the metrics tests to not use sos >>>>>>>>>>>>> > 2. disable the metrics tests >>>>>>>>>>>>> > 3. revert the sos patch until a decision is made on ^^ >>>>>>>>>>>>> >>>>>>>>>>>>> #3 is not an option, it would make Vdsm uninstallable on >>>>>>>>>>>>> newer RHEL >>>>>>>>>>>>> versions. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I've posted a patch https://gerrit.ovirt.org/100716 which is >>>>>>>>>>>> trying to install vdsm sos plugin if it's not installed either by vdsm nor >>>>>>>>>>>> sos. >>>>>>>>>>>> Currenlty waiting for CI, if run is successfull, I will extend >>>>>>>>>>>> the patch also for 4.3 basic suite. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> > Thanks, >>>>>>>>>>>>> > Dafna >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > ---------- Forwarded message --------- >>>>>>>>>>>>> > From: Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>>>>>> > Date: Mon, Jun 10, 2019 at 1:30 PM >>>>>>>>>>>>> > Subject: Re: [ OST Failure Report ] [ oVirt Master (vdsm) ] >>>>>>>>>>>>> [ 07-06-2019 ] >>>>>>>>>>>>> > [ 003_00_metrics_bootstrap.metrics_and_log_collector ] >>>>>>>>>>>>> > To: Martin Perina <mperina(a)redhat.com&gt;, Milan Zamazal < >>>>>>>>>>>>> mzamazal(a)redhat.com&gt;, >>>>>>>>>>>>> > Shirly Radco <sradco(a)redhat.com&gt; >>>>>>>>>>>>> > Cc: devel <devel(a)ovirt.org&gt;, infra <infra(a)ovirt.org&gt; >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > Shirly? any update on this? >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Fri, Jun 7, 2019 at 11:54 AM Dafna Ron <dron(a)redhat.com&gt; >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> >> Hi, >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> We have a failure in vdsm project on master. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> The issue is change: >>>>>>>>>>>>> >> https://gerrit.ovirt.org/#/c/100576/ - Remove SOS VDSM >>>>>>>>>>>>> plugin >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> which is failing on metrics as metrics is calling >>>>>>>>>>>>> sos-logcollector. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> The patch cannot be changed as until centos 7.7 when >>>>>>>>>>>>> sos-3.7-3, which >>>>>>>>>>>>> >> contains vdsm plugin will come out. >>>>>>>>>>>>> >> so until then, we are left with no sos plugin, which is >>>>>>>>>>>>> causing the >>>>>>>>>>>>> >> metrics test to fail. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Shirly, can you please take a look and see if we can >>>>>>>>>>>>> change the test to >>>>>>>>>>>>> >> not call sos-logcollector? >>>>>>>>>>>>> >> Please note, that we are expecting 4.3 to fail on same >>>>>>>>>>>>> issue very soon. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> failed job can be found here: >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14452/ >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> ERROR from test: >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> lago.ssh: DEBUG: Command 8626fe70 on >>>>>>>>>>>>> lago-basic-suite-master-engine errors: >>>>>>>>>>>>> >> ERROR: Failed to get a sosreport from: >>>>>>>>>>>>> lago-basic-suite-master-host-1; Could not parse sosreport output >>>>>>>>>>>>> >> ERROR: Failed to get a sosreport from: >>>>>>>>>>>>> lago-basic-suite-master-host-0; Could not parse sosreport output >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> lago.utils: DEBUG: Error while running thread Thread-3 >>>>>>>>>>>>> >> Traceback (most recent call last): >>>>>>>>>>>>> >> File "/usr/lib/python2.7/site-packages/lago/utils.py", >>>>>>>>>>>>> line 58, in _ret_via_queue >>>>>>>>>>>>> >> queue.put({'return': func()}) >>>>>>>>>>>>> >> File >>>>>>>>>>>>> "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_00_metrics_bootstrap.py", >>>>>>>>>>>>> line 97, in run_log_collector >>>>>>>>>>>>> >> 'log collector failed. Exit code is %s' % result.code >>>>>>>>>>>>> >> File "/usr/lib64/python2.7/unittest/case.py", line 462, >>>>>>>>>>>>> in assertTrue >>>>>>>>>>>>> >> raise self.failureException(msg) >>>>>>>>>>>>> >> AssertionError: log collector failed. Exit code is 1 >>>>>>>>>>>>> >> --------------------- >> end captured logging << ---------- >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Thanks, >>>>>>>>>>>>> >> Dafna >>>>>>>>>>>>> >> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Martin Perina >>>>>>>>>>>> Manager, Software Engineering >>>>>>>>>>>> Red Hat Czech s.r.o. >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>> List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/SQJDJHHQR52... >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>>> List Archives: >>>>>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GW3E7ZIE3WI... >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Eyal edri >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> MANAGER >>>>>>>>>>> >>>>>>>>>>> CONTINUOUS PRODUCTIZATION >>>>>>>>>>> >>>>>>>>>>> SYSTEM ENGINEERING >>>>>>>>>>> >>>>>>>>>>> Red Hat <https://www.redhat.com/> >>>>>>>>>>> <https://red.ht/sig> >>>>>>>>>>> phone: +972-9-7692018 >>>>>>>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ #cp-devel) >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/3TLL7C4LF76... >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Martin Perina >>>>>>>>> Manager, Software Engineering >>>>>>>>> Red Hat Czech s.r.o. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Martin Perina >>>>>>>> Manager, Software Engineering >>>>>>>> Red Hat Czech s.r.o. >>>>>>>> >>>>>>> >>>>> >>>>> -- >>>>> Martin Perina >>>>> Manager, Software Engineering >>>>> Red Hat Czech s.r.o. >>>>> >>>> > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. >