Re: [ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 14-09-2018 ] [ 002_bootstrap.add_hosts ]

On Mon, Sep 17, 2018 at 12:47 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > I think that in ovirt-engine we currently only build to centos. > since we have not had an engine build for 2 weeks (on master) I think we > should merge and worry about fc28 once it would be relevant. >

So let's get the change reverted and let's continue working on FC28 issue. Didi/Yuval/Sandro any issue with merging https://gerrit.ovirt.org/94357 ?

> the failure we have now could be another regression missed since the > project has been broken for two weeks. > > Thanks, > Dafna > > > > On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi(a)redhat.com&gt; > wrote: > >> On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >> > >> > Didi, Marin, any update on the patch? >> >> Yes - it passed. Actually failed, but only after host-deploy: >> >> >> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >> >> I'd rather not merge it as-is, because it will break fedora. >> >> If someone can have a look at the code generating the tar file, and can >> see if >> it's easy to make it work well for both centos and fedora, perhaps by >> explicitly >> setting all relevant params to some reasonable values, great. Otherwise, >> I guess >> we can merge for now, as fedora is still not supported anyway. >> >> Thanks, >> >> > >> > >> > On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi(a)redhat.com&gt; >> wrote: >> >> >> >> On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi(a)redhat.com&gt; >> wrote: >> >> > >> >> > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina(a)redhat.com&gt; >> wrote: >> >> > > >> >> > > >> >> > > >> >> > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < >> rnori(a)redhat.com&gt; wrote: >> >> > >> >> >> > >> I see the same errors on my dev env. From the logs attached by >> Andrej the response received by otopi has a bunch of null chars before the >> actual response CONFIRM DEPLOY_PROCEED=yes >> >> > >> >> >> > >> >> >> > >> >> >> > >> 2018-09-14 15:49:23,018+0200 DEBUG >> otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND >> ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED >> >> > >> >> >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes >> >> > > >> >> > > >> >> > > Didi/Sandro, could you please take a look? Below error seems like >> some issue in otopi, where an error is raised when handling binary input: >> >> > >> >> > Not sure the issue is "binary input" in general, but simply illegal >> >> > input. The prompt expects, as it says, one of these 3 replies: >> >> > >> >> > CONFIRM DEPLOY_PROCEED=yes >> >> > CONFIRM DEPLOY_PROCEED=no >> >> > ABORT DEPLOY_PROCEED >> >> > >> >> > Instead, judging from the file supplied by Andrej, it gets from the >> engine: >> >> > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes >> >> > >> >> > So either the engine now sends, for some reason, 7169 null bytes, in >> >> > this response, or there is some low-level change causing this to be >> >> > eventually supplied to otopi - a change in apache-sshd, openssh, >> some >> >> > library, the kernel, no idea. >> >> > >> >> > Well, thinking a bit, I have a wild guess: Perhaps it's related to >> the >> >> > patch introduced recently to change the tar blocking? >> >> >> >> https://gerrit.ovirt.org/94357 >> >> >> >> I am leaving soon, perhaps someone can try the manual job with the >> >> result of the check-patch job for above patch, to see if it fixes. >> >> Otherwise I'll do this tomorrow. >> >> >> >> > >> >> > > >> >> > > >> >> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context >> context._executeMethod:143 method exception >> >> > > Traceback (most recent call last): >> >> > > File "/usr/lib/python2.7/site-packages/otopi/context.py", line >> 133, in _executeMethod >> >> > > method['method']() >> >> > > File >> "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line >> 87, in _confirm >> >> > > prompt=True, >> >> > > File >> "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in >> confirm >> >> > > code=opcode, >> >> > > >> >> > > >> >> > >> >> >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron(a)redhat.com&gt; >> wrote: >> >> > >>> >> >> > >>> if you run it with mock you would remove any environmental >> conditions that can effect the outcome so I recommend using mock >> >> > >>> >> >> > >>> >> >> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < >> mperina(a)redhat.com&gt; wrote: >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron(a)redhat.com&gt; >> wrote: >> >> > >>>>> >> >> > >>>>> did you use mock to reproduce? >> >> > >>>> >> >> > >>>> >> >> > >>>> No, just run_suite under myself >> >> > >>>>> >> >> > >>>>> >> >> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < >> mperina(a)redhat.com&gt; wrote: >> >> > >>>>>> >> >> > >>>>>> Hi, >> >> > >>>>>> >> >> > >>>>>> the problem is that we haven't fetched the temporary >> host-deploy log from /tmp directory, so we don't know which string that >> host-deploy process sent to engine is causing that issue. I tried to >> reproduce on my local machine, but I was unable to reproduce it, >> 002_bootstrap phase finished successfully (other phases are still running). >> >> > >>>>>> >> >> > >>>>>> So if anyone is able to reproduce, please try to fetch >> host-deploy log from /tmp directory after the error is raised and share it. >> >> > >>>>>> >> >> > >>>>>> Thanks >> >> > >>>>>> >> >> > >>>>>> Martin >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron(a)redhat.com&gt; >> wrote: >> >> > >>>>>>> >> >> > >>>>>>> Full logs can be found here: >> >> > >>>>>>> >> >> > >>>>>>> >> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >> >> > >>>>>>> >> >> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < >> dron(a)redhat.com&gt; wrote: >> >> > >>>>>>>> >> >> > >>>>>>>> Hi, >> >> > >>>>>>>> >> >> > >>>>>>>> The previous regression was resolved and we now have a new >> regression. >> >> > >>>>>>>> >> >> > >>>>>>>> I don't think that the reported change is related so can >> someone from ovirt-engine take a look? >> >> > >>>>>>>> >> >> > >>>>>>>> The failure is add host on the upgrade suite. >> >> > >>>>>>>> >> >> > >>>>>>>> Please note that we have not had an engine-ovirt build for >> over 10 days due to several consecutive regressions and I would ask you to >> stop merging until we can stabilize the project and have a new package of >> engine. >> >> > >>>>>>>> >> >> > >>>>>>>> error: >> >> > >>>>>>>> >> >> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO >> [org.ovirt.engine.core.uutils.ssh.SSHDialog] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute >> 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; >> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap >> "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > >> /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && >> "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >> DIALOG/customization=bool:True' >> >> > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing >> Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >> >> > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) >> [5c91fcbd] transaction rolled back >> >> > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] >> Error during deploy dialog >> >> > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >> lago-upgrade-from-release-suite-master-host-0 install >> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >> VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during >> installation of Host lago-upgrade-from-release-suite-master-host-0: >> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >> exception is org.postgresql.util.PSQLException: ERROR: invalid byte >> sequence for encoding "UTF8": 0x00. >> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >> lago-upgrade-from-release-suite-master-host-0 install, preferring first >> exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; >> nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte >> sequence for encoding "UTF8": 0x00 >> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation >> failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', >> 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; >> SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte >> sequence for encoding "UTF8": 0x00; nested exception is >> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >> encoding "UTF8": 0x00 >> >> > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO >> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, >> SetVdsStatusVDSCommand(HostName = >> lago-upgrade-from-release-suite-master-host-0, >> SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', >> status='InstallFailed', nonOperationalReason='NONE', >> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >> >> > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO >> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, >> SetVdsStatusVDSCommand, return: , log id: 146cdc08 >> >> > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >> VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 >> installation failed. CallableStatementCallback; SQL [{call >> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >> encoding "UTF8": 0x00; nested exception is >> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >> encoding "UTF8": 0x00. >> >> > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO >> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object >> 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', >> sharedLocks=''}' >> >> > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO >> [org.ovirt.engine.core.bll.quota.QuotaManager] >> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. >> (19 msec) >> >> > >>>>>>>> (END) >> >> > >>>>>>>> >> >> > >>>>>>>> Thanks, >> >> > >>>>>>>> Dafna >> >> > >>>>>>>> >> >> > >>>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> -- >> >> > >>>>>> Martin Perina >> >> > >>>>>> Associate Manager, Software Engineering >> >> > >>>>>> Red Hat Czech s.r.o. >> >> > >>>>> >> >> > >>>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> -- >> >> > >>>> Martin Perina >> >> > >>>> Associate Manager, Software Engineering >> >> > >>>> Red Hat Czech s.r.o. >> >> > >>> >> >> > >>> >> >> > >> >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > Martin Perina >> >> > > Associate Manager, Software Engineering >> >> > > Red Hat Czech s.r.o. >> >> > >> >> > >> >> > >> >> > -- >> >> > Didi >> >> >> >> >> >> >> >> -- >> >> Didi >> >> >> >> -- >> Didi >> > -- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.