Re: [ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 14-09-2018 ] [ 002_bootstrap.add_hosts ]

didi and Sandro are both not working today.

if they are not back by Friday, can we try to find the patch that caused this and revert? CQ reported this as the cause: https://gerrit.ovirt.org/#/c/94081/3

But it did not seem related to this issue, however, it was merged 11 days ago and we could have missed this due to several regressions during these 10 days.

On Fri, Sep 14, 2018 at 4:06 PM, Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori(a)redhat.com&gt; > wrote: > >> I see the same errors on my dev env. From the logs attached by Andrej >> the response received by otopi has a bunch of null chars before the actual >> response CONFIRM DEPLOY_PROCEED=yes >> >> >> >> 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine >> dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM >> DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED >> >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes >> > > Didi/Sandro, could you please take a look? Below error seems like some > issue in otopi, where an error is raised when handling binary input: > > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context > context._executeMethod:143 method exception > > > Traceback (most recent call last): > > > > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in > _executeMethod > > method['method']() > > > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", > line 87, in _confirm > > prompt=True, > > > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", > line 478, in confirm > > code=opcode, > > > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron(a)redhat.com&gt; wrote: >> >>> if you run it with mock you would remove any environmental conditions >>> that can effect the outcome so I recommend using mock >>> >>> >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina(a)redhat.com&gt; >>> wrote: >>> >>>> >>>> >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>>> did you use mock to reproduce? >>>>> >>>> >>>> No, just run_suite under myself >>>> >>>>> >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> the problem is that we haven't fetched the temporary host-deploy log >>>>>> from /tmp directory, so we don't know which string that host-deploy process >>>>>> sent to engine is causing that issue. I tried to reproduce on my local >>>>>> machine, but I was unable to reproduce it, 002_bootstrap phase finished >>>>>> successfully (other phases are still running). >>>>>> >>>>>> So if anyone is able to reproduce, please try to fetch host-deploy >>>>>> log from /tmp directory after the error is raised and share it. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Martin >>>>>> >>>>>> >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>> >>>>>>> Full logs can be found here: >>>>>>> >>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi >>>>>>> rt-master_change-queue-tester/10307/artifact/upgrade-from-re >>>>>>> lease-suite.el7.x86_64/test_logs/upgrade-from-release-suite- >>>>>>> master/post-002_bootstrap.py/ >>>>>>> >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> The previous regression was resolved and we now have a new >>>>>>>> regression. >>>>>>>> >>>>>>>> I don't think that the reported change is related so can someone >>>>>>>> from ovirt-engine take a look? >>>>>>>> >>>>>>>> The failure is add host on the upgrade suite. >>>>>>>> >>>>>>>> Please note that we have not had an engine-ovirt build for over 10 >>>>>>>> days due to several consecutive regressions and I would ask you to stop >>>>>>>> merging until we can stabilize the project and have a new package of >>>>>>>> engine. >>>>>>>> >>>>>>>> error: >>>>>>>> >>>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] >>>>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute >>>>>>>> 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; >>>>>>>> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t >>>>>>>> ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm >>>>>>>> -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C >>>>>>>> "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >>>>>>>> DIALOG/customization=bool:True' >>>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb >>>>>>>> roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] >>>>>>>> EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host >>>>>>>> lago-upgrade-from-release-suite-master-host-0. Stage: >>>>>>>> Initializing. >>>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] >>>>>>>> (VdsDeploy) [5c91fcbd] transaction rolled back >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>>>>>>> (VdsDeploy) [5c91fcbd] Error during deploy dialog >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>>>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during >>>>>>>> host lago-upgrade-from-release-suite-master-host-0 install >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb >>>>>>>> roker.auditloghandling.AuditLogDirector] >>>>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >>>>>>>> VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during >>>>>>>> installation of Host lago-upgrade-from-release-suite-master-host-0: >>>>>>>> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >>>>>>>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>>>>> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >>>>>>>> exception is org.postgresql.util.PSQLException: ERROR: invalid >>>>>>>> byte sequence for encoding "UTF8": 0x00. >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>>>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during >>>>>>>> host lago-upgrade-from-release-suite-master-host-0 install, >>>>>>>> preferring first exception: CallableStatementCallback; SQL [{call >>>>>>>> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>>>>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >>>>>>>> encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: >>>>>>>> ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos >>>>>>>> tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) >>>>>>>> [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', >>>>>>>> 'lago-upgrade-from-release-suite-master-host-0': >>>>>>>> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >>>>>>>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>>>>> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >>>>>>>> exception is org.postgresql.util.PSQLException: ERROR: invalid >>>>>>>> byte sequence for encoding "UTF8": 0x00 >>>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, >>>>>>>> SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, >>>>>>>> SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', >>>>>>>> status='InstallFailed', nonOperationalReason='NONE', >>>>>>>> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, >>>>>>>> SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb >>>>>>>> roker.auditloghandling.AuditLogDirector] >>>>>>>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >>>>>>>> VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 >>>>>>>> installation failed. CallableStatementCallback; SQL [{call >>>>>>>> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>>>>>>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >>>>>>>> encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: >>>>>>>> ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos >>>>>>>> tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) >>>>>>>> [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ >>>>>>>> e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] >>>>>>>> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota >>>>>>>> Cache updated. (19 msec) >>>>>>>> (END) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dafna >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Martin Perina >>>>>> Associate Manager, Software Engineering >>>>>> Red Hat Czech s.r.o. >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Martin Perina >>>> Associate Manager, Software Engineering >>>> Red Hat Czech s.r.o. >>>> >>> >>> >> > > > -- > Martin Perina > Associate Manager, Software Engineering > Red Hat Czech s.r.o. >