Re: [ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 10-09-2018 ] [ 002_bootstrap.add_hosts, bootstrap.verify_add_hosts ]

Hi, we had a failure to deploy hosts for change; https://gerrit.ovirt.org/#/c/94228/1 I don't think its related directly to the change but anything between packaging: use system provided nimbus-jose-jwt also on el7 patch and current patch could have caused the failure. https://gerrit.ovirt.org/#/q/project:ovirt-engine+status: Merged+branch:master Sandro, Ondra, can you please have a look?

(Relevant) error snippet from the log: <error> 018-09-10 04:05:33,850-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] Installation of lago-upgrade-from-release-suite-master-host-0. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt- host-deploy.tar 2018-09-10 04:05:33,850-04 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh 2018-09-10 04:05:33,915-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-10 04:05:34,694-04 INFO [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [33bb53b] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-10 04:05:34,710-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [33bb53b] transaction rolled back 2018-09-10 04:05:34,719-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [33bb53b] Error during deploy dialog 2018-09-10 04:05:34,723-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-10 04:05:34,736-04 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-10 04:05:34,737-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-10 04:05:34,737-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] Host installation failed for host 'febee66d-fa4b-4321-9c99-cb053ddd4817', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-10 04:05:34,760-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='febee66d-fa4b-4321-9c99-cb053ddd4817', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 793ad1ed 2018-09-10 04:05:34,769-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] FINISH, SetVdsStatusVDSCommand, return: , log id: 793ad1ed 2018-09-10 04:05:34,781-04 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-10 04:05:34,794-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [33bb53b] Lock freed to object 'EngineLock:{exclusiveLocks='[febee66d-fa4b-4321-9c99-cb053ddd4817=VDS]', sharedLocks=''}' 2018-09-10 04:06:10,003-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (20 msec) (END) </error>