Re: [CQ]: 91908,6 (otopi) failed "ovirt-master" system tests

On Sun, Jun 24, 2018 at 1:25 PM, Barak Korren <bkorren(a)redhat.com&gt; wrote: > > > On 24 June 2018 at 13:01, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > >> On Mon, Jun 18, 2018 at 12:03 PM, Yedidyah Bar David <didi(a)redhat.com&gt; >> wrote: >> >>> On Mon, Jun 18, 2018 at 9:27 AM, Yedidyah Bar David <didi(a)redhat.com&gt; >>> wrote: >>> >>>> On Sun, Jun 17, 2018 at 6:21 PM, Barak Korren <bkorren(a)redhat.com&gt; >>>> wrote: >>>> >>>>> >>>>> >>>>> On 17 June 2018 at 16:33, Eyal Edri <eedri(a)redhat.com&gt; wrote: >>>>> >>>>>> Didn't we drop fcraw support from CI? >>>>>> >>>>>> No we did not. People are not using it, but we still allow creating >>>>> fcraw chroots, and check that if works. >>>>> >>>>> >>>>> >>>>>> On Sun, Jun 17, 2018 at 2:13 PM, Emil Natan <ena(a)redhat.com&gt; wrote: >>>>>> >>>>>>> build-artifacts failed for fcraw x86_64. >>>>>>> The reason is: >>>>>>> 11:39:42 Running scriptlet: java-1.8.0-openjdk-headless-1:1.8.0.172-7.b11.fc29.x >>>>>>> 1/1 >>>>>>> 11:39:42 error: lua script failed: [string >>>>>>> "%pretrans(java-1.8.0-openjdk-headless-1:1.8.0.172-7.b11.fc29.x86_64)"]:27: >>>>>>> attempt to concatenate a nil value (field 'path') >>>>>>> >>>>>>> This is an known issue, though the ticket was closed: >>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1451439 >>>>>>> >>>>>>> >>>> To clarify - it's not a "known issue" - it was reported in the past, >>>> but was not reproducible. >>>> >>>> Also to me, while it happened several times, it didn't happen always. >>>> >>>> I suspect it's some infra issue, but not too many details yet. >>>> >>>> There is a lua variable, package.path, which seems to not be >>>> initialized for some reason. >>>> >>>> Try this: >>>> >>>> lua -e 'print(package.path)' >>>> >>>> On my machine, it outputs: >>>> >>>> ./?.lua;/usr/share/lua/5.1/?.lua;/usr/share/lua/5.1/?/init.l >>>> ua;/usr/lib64/lua/5.1/?.lua;/usr/lib64/lua/5.1/?/init.lua >>>> >>>> But in CI, sometimes, it's 'nil'. This is what is causing above error. >>>> Why is it nil? No idea. I didn't yet look at how rpm inits/uses lua, but >>>> that seems like the next step... >>>> >>>> As Emil mentioned, I commented on the bug, but if CI people have a >>>> clue what might cause this (everything from yum caches to >>>> differently-inited system stuff like /run or whatever), please comment... >>>> >>> >>> Does not fail anymore - probably fixed by a 'dnf update' you did in the >>> mock chroot or something similar, not sure. Thanks. >>> >> >> Failed again [1] for the same reason: >> >> *09:21:55* Building remotely on vm0054.workers-phx.ovirt.org <https://jenkins.ovirt.org/computer/vm0054.workers-phx.ovirt.org> (libvirt phx fc28 nested) in workspace /home/jenkins/workspace/otopi_master_build-artifacts-fcraw-x86_64 >> ... >> *09:25:23* error: lua script failed: [string "%pretrans(java-1.8.0-openjdk-headless-1:1.8.0.172-9.b11.fc29.x86_64)"]:27: attempt to concatenate a nil value (field 'path') >> >> I guessed (above) that it was fixed in previous case by running dnf >> update. Can whoever that did that do this on all slaves? Thanks! >> >> [1] https://jenkins.ovirt.org/job/otopi_master_build-artifacts-f >> craw-x86_64/13/ >> >> > > nlo, you have the wrong mental image about how the slaves work. > (What's NLO?)

> > At the point that this is failing, your code is not running on the > slaves, its running inside its own chroot which was either clean installed > for it or extracted from a (at most) day old cache. > > Nothing we will do on the slave will help you with this, if the solution > for the issue is to have some newer package, you must run 'dnf update' for > it from the build script. > > Having said the above, looking at the error, it seems the issue is with > '%pretrans(java-1.8.0-openjdk-headless-1:1.8.0.172-9.b11.fc29.x86_64)' > which belongs to the newer package you are trying to upgrade to as opposed > to an older cached version. So it may be the case that you saw your build > passing on some slaves which still had some older DNF cache around. > > If that is the latest build of the package, you probably need to file a > bug against it to have the maintainers produce a better build. > Please check previous posts in this thread. It was already discussed to death. These builds failed on one of the slaves, then started working. I thought it's because something you (CI) did - 'dnf update', updating the cache, whatever - no idea. If you claim it's not, fine - is there then a way to cause a build to run on a specific slave? If so, we can try to run it again on that slave and see what happens.

The thread also mentioned a bug against rpm, and I also commented at that bug. I eventually gave up on it, mainly because it didn't reproduce anymore on that slave, and I thought it's "solved" for all slaves, until it now happened again.

> > > >> >>> >>>> >>>> Thanks, >>>> >>>> >>>>> Didi already commented on it. >>>>>>> >>>>>>> >>>>>>> On Sun, Jun 17, 2018 at 12:05 PM, oVirt Jenkins <jenkins(a)ovirt.org&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Change 91908,6 (otopi) is probably the reason behind recent system >>>>>>>> test >>>>>>>> failures in the "ovirt-master" change queue and needs to be fixed. >>>>>>>> >>>>>>>> This change had been removed from the testing queue. Artifacts >>>>>>>> build from this >>>>>>>> change will not be released until it is fixed. >>>>>>>> >>>>>>>> For further details about the change see: >>>>>>>> https://gerrit.ovirt.org/#/c/91908/6 >>>>>>>> >>>>>>>> For failed test results see: >>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste >>>>>>>> r/8239/ >>>>>>>> _______________________________________________ >>>>>>>> Infra mailing list -- infra(a)ovirt.org >>>>>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>>>>>> y/about/community-guidelines/ >>>>>>>> List Archives: https://lists.ovirt.org/archiv >>>>>>>> es/list/infra(a)ovirt.org/message/C2VUNTJ4GQ4TACZFXH7IMFR5EMR2ERAR/ >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Emil Natan >>>>>>> RHV DevOps >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Infra mailing list -- infra(a)ovirt.org >>>>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>>>>> y/about/community-guidelines/ >>>>>>> List Archives: https://lists.ovirt.org/archiv >>>>>>> es/list/infra(a)ovirt.org/message/WSYSKDQFZZZWKKI3JEKZNBRWQNSEAZZZ/ >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Eyal edri >>>>>> >>>>>> >>>>>> MANAGER >>>>>> >>>>>> RHV DevOps >>>>>> >>>>>> EMEA VIRTUALIZATION R&D >>>>>> >>>>>> >>>>>> Red Hat EMEA <https://www.redhat.com/> >>>>>> <https://red.ht/sig> TRIED. TESTED. TRUSTED. >>>>>> <https://redhat.com/trusted> >>>>>> phone: +972-9-7692018 >>>>>> irc: eedri (on #tlv #rhev-dev #rhev-integ) >>>>>> >>>>>> _______________________________________________ >>>>>> Infra mailing list -- infra(a)ovirt.org >>>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>>>> y/about/community-guidelines/ >>>>>> List Archives: https://lists.ovirt.org/archiv >>>>>> es/list/infra(a)ovirt.org/message/HCKQA7ZGATCPWPQD2Y4HNKAQBHNJWC5W/ >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Barak Korren >>>>> RHV DevOps team , RHCE, RHCi >>>>> Red Hat EMEA >>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted >>>>> >>>>> _______________________________________________ >>>>> Infra mailing list -- infra(a)ovirt.org >>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>>> y/about/community-guidelines/ >>>>> List Archives: https://lists.ovirt.org/archiv >>>>> es/list/infra(a)ovirt.org/message/RKX5BRAKQVU4TMX53KZA25WNECBDHZUA/ >>>>> >>>>> >>>> >>>> >>>> -- >>>> Didi >>>> >>> >>> >>> >>> -- >>> Didi >>> >> >> >> >> -- >> Didi >> > > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > -- Didi