Re: [ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 14-09-2018 ] [ 002_bootstrap.add_hosts ]

Ok, regarding the tar issue, there's another solution - since commons-compress hard coded the blocking factor to 1, while the default in tar is 20, we could continue creating the tar as we do today, but add a bunch of zeros to the end of the tarball. [yturgema@piggie ~/aa]$ ls -l ovirt-host-deploy_fc28.tar -rw-r--r--. 1 yturgema yturgema 2166272 Sep 2 17:03 ovirt-host-deploy_fc28.tar [yturgema@piggie ~/aa]$ python -c "print 10240-2166272%10240" 4608 [yturgema@piggie ~/aa]$ dd if=/dev/zero of=/dev/stdout bs=4608 count=1 | cat ovirt-host-deploy_fc28.tar - > new_host_deploy.tar 1+0 records in 1+0 records out 4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000373488 s, 12.3 MB/s [yturgema@piggie ~/aa]$ ls -l new_host_deploy.tar -rw-rw-r--. 1 yturgema yturgema 2170880 Sep 17 18:16 new_host_deploy.tar This would solve the problem, and not break el7, if this solution is acceptable, I can send a patch.

On Mon, Sep 17, 2018 at 6:02 PM, Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Mon, 17 Sep 2018, 16:25 Ravi Shankar Nori, <rnori(a)redhat.com&gt; wrote: > >> host-deploy is still broken on master fc28 >> > > Yes, there are multiple issues on FC28, but the question is if this fixed > OST on CentOS? > > >> On Mon, Sep 17, 2018 at 8:01 AM, Yuval Turgeman <yturgema(a)redhat.com&gt; >> wrote: >> >>> I'm pretty sure I verified this on el7 as well, i'll check again, but >>> thinking about it, tar will stop when it gets to the first empty block, so >>> if the record size on the engine's side is large and the end is filled with >>> zeros, -b1 will make it stop at the first empty block so the next read on >>> the host's side would get the trailing zeros which is what otopi reads. >>> Btw, it could be a problem with deployed el7 systems as well, if for >>> any reason the default on the host is set to something that is more than 20 >>> blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on >>> the host side). >>> It's ok to revert the patch to fix the regression, but I don't see any >>> other way other than -b1... perhaps add a `cat -` after to just read until >>> EOF or something, or have otopi strip the input. >>> >>> On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth(a)redhat.com&gt; >>> wrote: >>> >>>> Didi, >>>> >>>> Is this what you are looking for >>>> https://ovirt-jira.atlassian.net/browse/OVIRT-2259 >>>> ? >>>> Galit >>>> >>>> On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>>> I think that in ovirt-engine we currently only build to centos. >>>>> since we have not had an engine build for 2 weeks (on master) I think >>>>> we should merge and worry about fc28 once it would be relevant. >>>>> >>>>> the failure we have now could be another regression missed since the >>>>> project has been broken for two weeks. >>>>> >>>>> Thanks, >>>>> Dafna >>>>> >>>>> >>>>> >>>>> On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>> > >>>>>> > Didi, Marin, any update on the patch? >>>>>> >>>>>> Yes - it passed. Actually failed, but only after host-deploy: >>>>>> >>>>>> >>>>>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-te... >>>>>> >>>>>> I'd rather not merge it as-is, because it will break fedora. >>>>>> >>>>>> If someone can have a look at the code generating the tar file, and >>>>>> can see if >>>>>> it's easy to make it work well for both centos and fedora, perhaps >>>>>> by explicitly >>>>>> setting all relevant params to some reasonable values, great. >>>>>> Otherwise, I guess >>>>>> we can merge for now, as fedora is still not supported anyway. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> > >>>>>> > >>>>>> > On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David < >>>>>> didi(a)redhat.com&gt; wrote: >>>>>> >> >>>>>> >> On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David < >>>>>> didi(a)redhat.com&gt; wrote: >>>>>> >> > >>>>>> >> > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina < >>>>>> mperina(a)redhat.com&gt; wrote: >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < >>>>>> rnori(a)redhat.com&gt; wrote: >>>>>> >> > >> >>>>>> >> > >> I see the same errors on my dev env. From the logs attached >>>>>> by Andrej the response received by otopi has a bunch of null chars before >>>>>> the actual response CONFIRM DEPLOY_PROCEED=yes >>>>>> >> > >> >>>>>> >> > >> >>>>>> >> > >> >>>>>> >> > >> 2018-09-14 15:49:23,018+0200 DEBUG >>>>>> otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND >>>>>> ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED >>>>>> >> > >> >>>>>> >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > Didi/Sandro, could you please take a look? Below error seems >>>>>> like some issue in otopi, where an error is raised when handling binary >>>>>> input: >>>>>> >> > >>>>>> >> > Not sure the issue is "binary input" in general, but simply >>>>>> illegal >>>>>> >> > input. The prompt expects, as it says, one of these 3 replies: >>>>>> >> > >>>>>> >> > CONFIRM DEPLOY_PROCEED=yes >>>>>> >> > CONFIRM DEPLOY_PROCEED=no >>>>>> >> > ABORT DEPLOY_PROCEED >>>>>> >> > >>>>>> >> > Instead, judging from the file supplied by Andrej, it gets from >>>>>> the engine: >>>>>> >> > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes >>>>>> >> > >>>>>> >> > So either the engine now sends, for some reason, 7169 null >>>>>> bytes, in >>>>>> >> > this response, or there is some low-level change causing this >>>>>> to be >>>>>> >> > eventually supplied to otopi - a change in apache-sshd, >>>>>> openssh, some >>>>>> >> > library, the kernel, no idea. >>>>>> >> > >>>>>> >> > Well, thinking a bit, I have a wild guess: Perhaps it's related >>>>>> to the >>>>>> >> > patch introduced recently to change the tar blocking? >>>>>> >> >>>>>> >> https://gerrit.ovirt.org/94357 >>>>>> >> >>>>>> >> I am leaving soon, perhaps someone can try the manual job with the >>>>>> >> result of the check-patch job for above patch, to see if it fixes. >>>>>> >> Otherwise I'll do this tomorrow. >>>>>> >> >>>>>> >> > >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context >>>>>> context._executeMethod:143 method exception >>>>>> >> > > Traceback (most recent call last): >>>>>> >> > > File "/usr/lib/python2.7/site-packages/otopi/context.py", >>>>>> line 133, in _executeMethod >>>>>> >> > > method['method']() >>>>>> >> > > File >>>>>> "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line >>>>>> 87, in _confirm >>>>>> >> > > prompt=True, >>>>>> >> > > File >>>>>> "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in >>>>>> confirm >>>>>> >> > > code=opcode, >>>>>> >> > > >>>>>> >> > > >>>>>> >> > >> >>>>>> >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron(a)redhat.com&gt; >>>>>> wrote: >>>>>> >> > >>> >>>>>> >> > >>> if you run it with mock you would remove any environmental >>>>>> conditions that can effect the outcome so I recommend using mock >>>>>> >> > >>> >>>>>> >> > >>> >>>>>> >> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < >>>>>> mperina(a)redhat.com&gt; wrote: >>>>>> >> > >>>> >>>>>> >> > >>>> >>>>>> >> > >>>> >>>>>> >> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron < >>>>>> dron(a)redhat.com&gt; wrote: >>>>>> >> > >>>>> >>>>>> >> > >>>>> did you use mock to reproduce? >>>>>> >> > >>>> >>>>>> >> > >>>> >>>>>> >> > >>>> No, just run_suite under myself >>>>>> >> > >>>>> >>>>>> >> > >>>>> >>>>>> >> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < >>>>>> mperina(a)redhat.com&gt; wrote: >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> Hi, >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> the problem is that we haven't fetched the temporary >>>>>> host-deploy log from /tmp directory, so we don't know which string that >>>>>> host-deploy process sent to engine is causing that issue. I tried to >>>>>> reproduce on my local machine, but I was unable to reproduce it, >>>>>> 002_bootstrap phase finished successfully (other phases are still running). >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> So if anyone is able to reproduce, please try to fetch >>>>>> host-deploy log from /tmp directory after the error is raised and share it. >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> Thanks >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> Martin >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron < >>>>>> dron(a)redhat.com&gt; wrote: >>>>>> >> > >>>>>>> >>>>>> >> > >>>>>>> Full logs can be found here: >>>>>> >> > >>>>>>> >>>>>> >> > >>>>>>> >>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >>>>>> >> > >>>>>>> >>>>>> >> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < >>>>>> dron(a)redhat.com&gt; wrote: >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>>> Hi, >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>>> The previous regression was resolved and we now have a >>>>>> new regression. >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>>> I don't think that the reported change is related so >>>>>> can someone from ovirt-engine take a look? >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>>> The failure is add host on the upgrade suite. >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>>> Please note that we have not had an engine-ovirt build >>>>>> for over 10 days due to several consecutive regressions and I would ask you >>>>>> to stop merging until we can stabilize the project and have a new package >>>>>> of engine. >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>>> error: >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO >>>>>> [org.ovirt.engine.core.uutils.ssh.SSHDialog] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute >>>>>> 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; >>>>>> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap >>>>>> "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > >>>>>> /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && >>>>>> "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >>>>>> DIALOG/customization=bool:True' >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO >>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>> (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing >>>>>> Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO >>>>>> [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) >>>>>> [5c91fcbd] transaction rolled back >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR >>>>>> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] >>>>>> Error during deploy dialog >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR >>>>>> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >>>>>> lago-upgrade-from-release-suite-master-host-0 install >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >>>>>> VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during >>>>>> installation of Host lago-upgrade-from-release-suite-master-host-0: >>>>>> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >>>>>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>>> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >>>>>> exception is org.postgresql.util.PSQLException: ERROR: invalid byte >>>>>> sequence for encoding "UTF8": 0x00. >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >>>>>> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >>>>>> lago-upgrade-from-release-suite-master-host-0 install, preferring first >>>>>> exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, >>>>>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>>> ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; >>>>>> nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte >>>>>> sequence for encoding "UTF8": 0x00 >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >>>>>> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation >>>>>> failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', >>>>>> 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; >>>>>> SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte >>>>>> sequence for encoding "UTF8": 0x00; nested exception is >>>>>> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >>>>>> encoding "UTF8": 0x00 >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO >>>>>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, >>>>>> SetVdsStatusVDSCommand(HostName = >>>>>> lago-upgrade-from-release-suite-master-host-0, >>>>>> SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', >>>>>> status='InstallFailed', nonOperationalReason='NONE', >>>>>> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO >>>>>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, >>>>>> SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR >>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >>>>>> VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 >>>>>> installation failed. CallableStatementCallback; SQL [{call >>>>>> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >>>>>> encoding "UTF8": 0x00; nested exception is >>>>>> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >>>>>> encoding "UTF8": 0x00. >>>>>> >> > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO >>>>>> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object >>>>>> 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', >>>>>> sharedLocks=''}' >>>>>> >> > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO >>>>>> [org.ovirt.engine.core.bll.quota.QuotaManager] >>>>>> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. >>>>>> (19 msec) >>>>>> >> > >>>>>>>> (END) >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>>> Thanks, >>>>>> >> > >>>>>>>> Dafna >>>>>> >> > >>>>>>>> >>>>>> >> > >>>>>>> >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> >>>>>> >> > >>>>>> -- >>>>>> >> > >>>>>> Martin Perina >>>>>> >> > >>>>>> Associate Manager, Software Engineering >>>>>> >> > >>>>>> Red Hat Czech s.r.o. >>>>>> >> > >>>>> >>>>>> >> > >>>>> >>>>>> >> > >>>> >>>>>> >> > >>>> >>>>>> >> > >>>> >>>>>> >> > >>>> -- >>>>>> >> > >>>> Martin Perina >>>>>> >> > >>>> Associate Manager, Software Engineering >>>>>> >> > >>>> Red Hat Czech s.r.o. >>>>>> >> > >>> >>>>>> >> > >>> >>>>>> >> > >> >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > -- >>>>>> >> > > Martin Perina >>>>>> >> > > Associate Manager, Software Engineering >>>>>> >> > > Red Hat Czech s.r.o. >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > -- >>>>>> >> > Didi >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> -- >>>>>> >> Didi >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Didi >>>>>> >>>>> _______________________________________________ >>>>> Infra mailing list -- infra(a)ovirt.org >>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CG2IYPXSSEF... >>>>> >>>> >>>> >>>> -- >>>> >>>> GALIT ROSENTHAL >>>> >>>> SOFTWARE ENGINEER >>>> >>>> Red Hat >>>> >>>> <https://www.redhat.com/> >>>> >>>> galit(a)gmail.com T: 972-9-7692230 >>>> <https://red.ht/sig> >>>> >>> >>> >>> _______________________________________________ >>> Infra mailing list -- infra(a)ovirt.org >>> To unsubscribe send an email to infra-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/QMRM2INTCRD... >>> >>> >> _______________________________________________ Infra mailing list -- infra(a)ovirt.org To unsubscribe send an email to infra-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7MG4ZPPXOVM...