[ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 14-09-2018 ] [ 002_bootstrap.add_hosts ]
Hi, The previous regression was resolved and we now have a new regression. I don't think that the reported change is related so can someone from ovirt-engine take a look? The failure is add host on the upgrade suite. Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. error: 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END) Thanks, Dafna
Full logs can be found here: https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
The previous regression was resolved and we now have a new regression.
I don't think that the reported change is related so can someone from ovirt-engine take a look?
The failure is add host on the upgrade suite.
Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine.
error:
2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END)
Thanks, Dafna
Hi, the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. Thanks Martin On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/ job/ovirt-master_change-queue-tester/10307/artifact/upgrade- from-release-suite.el7.x86_64/test_logs/upgrade-from- release-suite-master/post-002_bootstrap.py/
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
The previous regression was resolved and we now have a new regression.
I don't think that the reported change is related so can someone from ovirt-engine take a look?
The failure is add host on the upgrade suite.
Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine.
error:
2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END)
Thanks, Dafna
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
did you use mock to reproduce? On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ ovirt-master_change-queue-tester/10307/artifact/upgrade-from -release-suite.el7.x86_64/test_logs/upgrade-from-release- suite-master/post-002_bootstrap.py/
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
The previous regression was resolved and we now have a new regression.
I don't think that the reported change is related so can someone from ovirt-engine take a look?
The failure is add host on the upgrade suite.
Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine.
error:
2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END)
Thanks, Dafna
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
Hi, I hit this problem on a development engine. If I compile the same engine code to rpm, install it on a VM, and then try to add the same host, it works. Attaching logs from /tmp. Andrej On Fri, 14 Sep 2018 at 15:49, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change...
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
The previous regression was resolved and we now have a new regression.
I don't think that the reported change is related so can someone from ovirt-engine take a look?
The failure is add host on the upgrade suite.
Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine.
error:
2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END)
Thanks, Dafna
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YHL3H3BNVE6R3D...
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi rt-master_change-queue-tester/10307/artifact/upgrade-from- release-suite.el7.x86_64/test_logs/upgrade-from-release-suit e-master/post-002_bootstrap.py/
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
The previous regression was resolved and we now have a new regression.
I don't think that the reported change is related so can someone from ovirt-engine take a look?
The failure is add host on the upgrade suite.
Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine.
error:
2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END)
Thanks, Dafna
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi rt-master_change-queue-tester/10307/artifact/upgrade-from-re lease-suite.el7.x86_64/test_logs/upgrade-from-release-suite- master/post-002_bootstrap.py/
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
The previous regression was resolved and we now have a new regression.
I don't think that the reported change is related so can someone from ovirt-engine take a look?
The failure is add host on the upgrade suite.
Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine.
error:
2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END)
Thanks, Dafna
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi rt-master_change-queue-tester/10307/artifact/upgrade-from-re lease-suite.el7.x86_64/test_logs/upgrade-from-release-suite- master/post-002_bootstrap.py/
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
The previous regression was resolved and we now have a new regression.
I don't think that the reported change is related so can someone from ovirt-engine take a look?
The failure is add host on the upgrade suite.
Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine.
error:
2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END)
Thanks, Dafna
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori@redhat.com> wrote:
I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes
2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED
^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input: 2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm prompt=True, File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm code=opcode,
On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi rt-master_change-queue-tester/10307/artifact/upgrade-from-re lease-suite.el7.x86_64/test_logs/upgrade-from-release-suite- master/post-002_bootstrap.py/
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
> Hi, > > The previous regression was resolved and we now have a new > regression. > > I don't think that the reported change is related so can someone > from ovirt-engine take a look? > > The failure is add host on the upgrade suite. > > Please note that we have not had an engine-ovirt build for over 10 > days due to several consecutive regressions and I would ask you to stop > merging until we can stabilize the project and have a new package of > engine. > > error: > > 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute > 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; > MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; > trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && > "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine > DIALOG/customization=bool:True' > 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb > roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] > EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host > lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. > 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (VdsDeploy) [5c91fcbd] transaction rolled back > 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (VdsDeploy) [5c91fcbd] Error during deploy dialog > 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during > host lago-upgrade-from-release-suite-master-host-0 install > 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb > roker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: > VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during > installation of Host lago-upgrade-from-release-suite-master-host-0: > CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested > exception is org.postgresql.util.PSQLException: ERROR: invalid byte > sequence for encoding "UTF8": 0x00. > 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during > host lago-upgrade-from-release-suite-master-host-0 install, > preferring first exception: CallableStatementCallback; SQL [{call > insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for > encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: > ERROR: invalid byte sequence for encoding "UTF8": 0x00 > 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos > tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) > [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', > 'lago-upgrade-from-release-suite-master-host-0': > CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested > exception is org.postgresql.util.PSQLException: ERROR: invalid byte > sequence for encoding "UTF8": 0x00 > 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, > SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, > SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', > status='InstallFailed', nonOperationalReason='NONE', > stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 > 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, > SetVdsStatusVDSCommand, return: , log id: 146cdc08 > 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb > roker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: > VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 > installation failed. CallableStatementCallback; SQL [{call > insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for > encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: > ERROR: invalid byte sequence for encoding "UTF8": 0x00. > 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos > tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) > [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ > e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' > 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache > updated. (19 msec) > (END) > > Thanks, > Dafna > >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
didi and Sandro are both not working today. if they are not back by Friday, can we try to find the patch that caused this and revert? CQ reported this as the cause: https://gerrit.ovirt.org/#/c/94081/3 But it did not seem related to this issue, however, it was merged 11 days ago and we could have missed this due to several regressions during these 10 days. On Fri, Sep 14, 2018 at 4:06 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori@redhat.com> wrote:
I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes
2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED
^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
method['method']()
File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm
prompt=True,
File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm
code=opcode,
On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
> Full logs can be found here: > > https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi > rt-master_change-queue-tester/10307/artifact/upgrade-from-re > lease-suite.el7.x86_64/test_logs/upgrade-from-release-suite- > master/post-002_bootstrap.py/ > > On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: > >> Hi, >> >> The previous regression was resolved and we now have a new >> regression. >> >> I don't think that the reported change is related so can someone >> from ovirt-engine take a look? >> >> The failure is add host on the upgrade suite. >> >> Please note that we have not had an engine-ovirt build for over 10 >> days due to several consecutive regressions and I would ask you to stop >> merging until we can stabilize the project and have a new package of >> engine. >> >> error: >> >> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute >> 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; >> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; >> trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > >> /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && >> "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >> DIALOG/customization=bool:True' >> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] >> EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host >> lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] >> (VdsDeploy) [5c91fcbd] transaction rolled back >> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (VdsDeploy) [5c91fcbd] Error during deploy dialog >> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during >> host lago-upgrade-from-release-suite-master-host-0 install >> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >> VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during >> installation of Host lago-upgrade-from-release-suite-master-host-0: >> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >> exception is org.postgresql.util.PSQLException: ERROR: invalid >> byte sequence for encoding "UTF8": 0x00. >> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during >> host lago-upgrade-from-release-suite-master-host-0 install, >> preferring first exception: CallableStatementCallback; SQL [{call >> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >> encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: >> ERROR: invalid byte sequence for encoding "UTF8": 0x00 >> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos >> tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) >> [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', >> 'lago-upgrade-from-release-suite-master-host-0': >> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >> exception is org.postgresql.util.PSQLException: ERROR: invalid >> byte sequence for encoding "UTF8": 0x00 >> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, >> SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, >> SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', >> status='InstallFailed', nonOperationalReason='NONE', >> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, >> SetVdsStatusVDSCommand, return: , log id: 146cdc08 >> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >> VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 >> installation failed. CallableStatementCallback; SQL [{call >> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >> encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: >> ERROR: invalid byte sequence for encoding "UTF8": 0x00. >> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos >> tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) >> [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ >> e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] >> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache >> updated. (19 msec) >> (END) >> >> Thanks, >> Dafna >> >> >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
On Fri, Sep 14, 2018 at 5:15 PM, Dafna Ron <dron@redhat.com> wrote:
didi and Sandro are both not working today.
So we will need to wait until Sunday/Monday, because I don't think anyone else from the team really understand otopi internals.
if they are not back by Friday, can we try to find the patch that caused this and revert?
CQ reported this as the cause: https://gerrit.ovirt.org/#/c/94081/3
This is definitely not related, it has nothing to do with adding host
But it did not seem related to this issue, however, it was merged 11 days ago and we could have missed this due to several regressions during these 10 days.
On Fri, Sep 14, 2018 at 4:06 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori@redhat.com> wrote:
I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes
2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED
^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
method['method']()
File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm
prompt=True,
File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm
code=opcode,
On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
> Hi, > > the problem is that we haven't fetched the temporary host-deploy log > from /tmp directory, so we don't know which string that host-deploy process > sent to engine is causing that issue. I tried to reproduce on my local > machine, but I was unable to reproduce it, 002_bootstrap phase finished > successfully (other phases are still running). > > So if anyone is able to reproduce, please try to fetch host-deploy > log from /tmp directory after the error is raised and share it. > > Thanks > > Martin > > > On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: > >> Full logs can be found here: >> >> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi >> rt-master_change-queue-tester/10307/artifact/upgrade-from-re >> lease-suite.el7.x86_64/test_logs/upgrade-from-release-suite- >> master/post-002_bootstrap.py/ >> >> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> >> wrote: >> >>> Hi, >>> >>> The previous regression was resolved and we now have a new >>> regression. >>> >>> I don't think that the reported change is related so can someone >>> from ovirt-engine take a look? >>> >>> The failure is add host on the upgrade suite. >>> >>> Please note that we have not had an engine-ovirt build for over 10 >>> days due to several consecutive regressions and I would ask you to stop >>> merging until we can stabilize the project and have a new package of >>> engine. >>> >>> error: >>> >>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute >>> 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; >>> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t >>> ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm >>> -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C >>> "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >>> DIALOG/customization=bool:True' >>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb >>> roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] >>> EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host >>> lago-upgrade-from-release-suite-master-host-0. Stage: >>> Initializing. >>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] >>> (VdsDeploy) [5c91fcbd] transaction rolled back >>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>> (VdsDeploy) [5c91fcbd] Error during deploy dialog >>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during >>> host lago-upgrade-from-release-suite-master-host-0 install >>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb >>> roker.auditloghandling.AuditLogDirector] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >>> VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during >>> installation of Host lago-upgrade-from-release-suite-master-host-0: >>> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >>> exception is org.postgresql.util.PSQLException: ERROR: invalid >>> byte sequence for encoding "UTF8": 0x00. >>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during >>> host lago-upgrade-from-release-suite-master-host-0 install, >>> preferring first exception: CallableStatementCallback; SQL [{call >>> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >>> encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: >>> ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos >>> tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) >>> [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', >>> 'lago-upgrade-from-release-suite-master-host-0': >>> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >>> exception is org.postgresql.util.PSQLException: ERROR: invalid >>> byte sequence for encoding "UTF8": 0x00 >>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, >>> SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, >>> SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', >>> status='InstallFailed', nonOperationalReason='NONE', >>> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, >>> SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb >>> roker.auditloghandling.AuditLogDirector] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >>> VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 >>> installation failed. CallableStatementCallback; SQL [{call >>> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >>> encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: >>> ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos >>> tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) >>> [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ >>> e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] >>> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota >>> Cache updated. (19 msec) >>> (END) >>> >>> Thanks, >>> Dafna >>> >>> >> > > > -- > Martin Perina > Associate Manager, Software Engineering > Red Hat Czech s.r.o. >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori@redhat.com> wrote:
I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes
2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED
^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies: CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea. Well, thinking a bit, I have a wild guess: Perhaps it's related to the patch introduced recently to change the tar blocking?
2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm prompt=True, File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm code=opcode,
On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: > > Full logs can be found here: > > https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... > > On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: >> >> Hi, >> >> The previous regression was resolved and we now have a new regression. >> >> I don't think that the reported change is related so can someone from ovirt-engine take a look? >> >> The failure is add host on the upgrade suite. >> >> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >> >> error: >> >> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >> (END) >> >> Thanks, >> Dafna >> >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Didi
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori@redhat.com> wrote:
I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes
2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED
^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies:
CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED
Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes
So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea.
Well, thinking a bit, I have a wild guess: Perhaps it's related to the patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357 I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm prompt=True, File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm code=opcode,
On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote: > > Hi, > > the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). > > So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. > > Thanks > > Martin > > > On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: >> >> Full logs can be found here: >> >> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >> >> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: >>> >>> Hi, >>> >>> The previous regression was resolved and we now have a new regression. >>> >>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >>> >>> The failure is add host on the upgrade suite. >>> >>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >>> >>> error: >>> >>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >>> (END) >>> >>> Thanks, >>> Dafna >>> >> > > > > -- > Martin Perina > Associate Manager, Software Engineering > Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Didi
-- Didi
Didi, Marin, any update on the patch? On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com>
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori@redhat.com>
wrote:
I see the same errors on my dev env. From the logs attached by Andrej
2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine
wrote: the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED
^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies:
CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED
Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes
So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea.
Well, thinking a bit, I have a wild guess: Perhaps it's related to the patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
2018-09-14 15:49:23,032+0200 DEBUG otopi.context
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm prompt=True, File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py",
code=opcode,
On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental
conditions that can effect the outcome so I recommend using mock
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: > > did you use mock to reproduce?
No, just run_suite under myself > > > On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com>
wrote:
>> >> Hi, >> >> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy
context._executeMethod:143 method exception line 478, in confirm process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
>> >> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >> >> Thanks >> >> Martin >> >> >> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: >>> >>> Full logs can be found here: >>> >>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>> >>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: >>>> >>>> Hi, >>>> >>>> The previous regression was resolved and we now have a new regression. >>>> >>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >>>> >>>> The failure is add host on the upgrade suite. >>>> >>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >>>> >>>> error: >>>> >>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >>>> (END) >>>> >>>> Thanks, >>>> Dafna >>>> >>> >> >> >> >> -- >> Martin Perina >> Associate Manager, Software Engineering >> Red Hat Czech s.r.o. > >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Didi
-- Didi
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy: https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... I'd rather not merge it as-is, because it will break fedora. If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway. Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori@redhat.com> wrote:
I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes
2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED
^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies:
CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED
Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes
So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea.
Well, thinking a bit, I have a wild guess: Perhaps it's related to the patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm prompt=True, File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm code=opcode,
On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote: > > > > On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: >> >> did you use mock to reproduce? > > > No, just run_suite under myself >> >> >> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote: >>> >>> Hi, >>> >>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). >>> >>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >>> >>> Thanks >>> >>> Martin >>> >>> >>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: >>>> >>>> Full logs can be found here: >>>> >>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>>> >>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: >>>>> >>>>> Hi, >>>>> >>>>> The previous regression was resolved and we now have a new regression. >>>>> >>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >>>>> >>>>> The failure is add host on the upgrade suite. >>>>> >>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >>>>> >>>>> error: >>>>> >>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >>>>> (END) >>>>> >>>>> Thanks, >>>>> Dafna >>>>> >>>> >>> >>> >>> >>> -- >>> Martin Perina >>> Associate Manager, Software Engineering >>> Red Hat Czech s.r.o. >> >> > > > > -- > Martin Perina > Associate Manager, Software Engineering > Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Didi
-- Didi
-- Didi
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant. the failure we have now could be another regression missed since the project has been broken for two weeks. Thanks, Dafna On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com>
wrote:
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <
rnori@redhat.com> wrote:
I see the same errors on my dev env. From the logs attached by
Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes
2018-09-14 15:49:23,018+0200 DEBUG
otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED
^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies:
CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED
Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes
So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea.
Well, thinking a bit, I have a wild guess: Perhaps it's related to the patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
2018-09-14 15:49:23,032+0200 DEBUG otopi.context
context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm prompt=True, File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm code=opcode,
On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com>
wrote:
> > if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock > > > On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: >> >> >> >> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: >>> >>> did you use mock to reproduce? >> >> >> No, just run_suite under myself >>> >>> >>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: >>>> >>>> Hi, >>>> >>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). >>>> >>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >>>> >>>> Thanks >>>> >>>> Martin >>>> >>>> >>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: >>>>> >>>>> Full logs can be found here: >>>>> >>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>>>> >>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> The previous regression was resolved and we now have a new regression. >>>>>> >>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >>>>>> >>>>>> The failure is add host on the upgrade suite. >>>>>> >>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >>>>>> >>>>>> error: >>>>>> >>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >>>>>> (END) >>>>>> >>>>>> Thanks, >>>>>> Dafna >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Martin Perina >>>> Associate Manager, Software Engineering >>>> Red Hat Czech s.r.o. >>> >>> >> >> >> >> -- >> Martin Perina >> Associate Manager, Software Engineering >> Red Hat Czech s.r.o. > >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Didi
-- Didi
-- Didi
On Mon, Sep 17, 2018 at 12:47 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
So let's get the change reverted and let's continue working on FC28 issue. Didi/Yuval/Sandro any issue with merging https://gerrit.ovirt.org/94357 ?
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com>
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <
rnori@redhat.com> wrote:
> > I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes > > > > 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED > > ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies:
CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED
Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes
So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea.
Well, thinking a bit, I have a wild guess: Perhaps it's related to
wrote: the
patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
2018-09-14 15:49:23,032+0200 DEBUG otopi.context
context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm prompt=True, File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm code=opcode,
> > On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: >> >> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock >> >> >> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: >>> >>> >>> >>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: >>>> >>>> did you use mock to reproduce? >>> >>> >>> No, just run_suite under myself >>>> >>>> >>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: >>>>> >>>>> Hi, >>>>> >>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). >>>>> >>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >>>>> >>>>> Thanks >>>>> >>>>> Martin >>>>> >>>>> >>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: >>>>>> >>>>>> Full logs can be found here: >>>>>> >>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>>>>> >>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> The previous regression was resolved and we now have a new regression. >>>>>>> >>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >>>>>>> >>>>>>> The failure is add host on the upgrade suite. >>>>>>> >>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >>>>>>> >>>>>>> error: >>>>>>> >>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >>>>>>> (END) >>>>>>> >>>>>>> Thanks, >>>>>>> Dafna >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Martin Perina >>>>> Associate Manager, Software Engineering >>>>> Red Hat Czech s.r.o. >>>> >>>> >>> >>> >>> >>> -- >>> Martin Perina >>> Associate Manager, Software Engineering >>> Red Hat Czech s.r.o. >> >> >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Didi
-- Didi
-- Didi
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
Il giorno lun 17 set 2018 alle ore 13:27 Martin Perina <mperina@redhat.com> ha scritto:
On Mon, Sep 17, 2018 at 12:47 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
We are bui8lding for Fedora 28 and we care for it. That said, we can't introduce regression in el7 for supporting Fedora.
So let's get the change reverted and let's continue working on FC28 issue.
Didi/Yuval/Sandro any issue with merging https://gerrit.ovirt.org/94357 ?
I'm merging this one. We'll need a different solution.
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com>
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com>
wrote:
> > > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < rnori@redhat.com> wrote: >> >> I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes >> >> >> >> 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED >> >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes > > > Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies:
CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED
Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes
So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea.
Well, thinking a bit, I have a wild guess: Perhaps it's related to
wrote: the
patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context
context._executeMethod:143 method exception
> Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod > method['method']() > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm > prompt=True, > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm > code=opcode, > > >> >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: >>> >>> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock >>> >>> >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: >>>> >>>> >>>> >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: >>>>> >>>>> did you use mock to reproduce? >>>> >>>> >>>> No, just run_suite under myself >>>>> >>>>> >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). >>>>>> >>>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Martin >>>>>> >>>>>> >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: >>>>>>> >>>>>>> Full logs can be found here: >>>>>>> >>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>>>>>> >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < dron@redhat.com> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> The previous regression was resolved and we now have a new regression. >>>>>>>> >>>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >>>>>>>> >>>>>>>> The failure is add host on the upgrade suite. >>>>>>>> >>>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >>>>>>>> >>>>>>>> error: >>>>>>>> >>>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >>>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >>>>>>>> (END) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dafna >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Martin Perina >>>>>> Associate Manager, Software Engineering >>>>>> Red Hat Czech s.r.o. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Martin Perina >>>> Associate Manager, Software Engineering >>>> Red Hat Czech s.r.o. >>> >>> >> > > > > -- > Martin Perina > Associate Manager, Software Engineering > Red Hat Czech s.r.o.
-- Didi
-- Didi
-- Didi
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig> <https://www.redhat.com/en/events/red-hat-open-source-day-italia?sc_cid=701f2000000RgRyAAK>
On Mon, Sep 17, 2018 at 2:29 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno lun 17 set 2018 alle ore 13:27 Martin Perina <mperina@redhat.com> ha scritto:
On Mon, Sep 17, 2018 at 12:47 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
We are bui8lding for Fedora 28 and we care for it. That said, we can't introduce regression in el7 for supporting Fedora.
So let's get the change reverted and let's continue working on FC28 issue.
Didi/Yuval/Sandro any issue with merging https://gerrit.ovirt.org/94357 ?
I'm merging this one. We'll need a different solution.
Not that important, but you could have changed the subject line :-)
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com> wrote: > > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com> wrote: > > > > > > > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <rnori@redhat.com> wrote: > >> > >> I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes > >> > >> > >> > >> 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED > >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes > > > > > > Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input: > > Not sure the issue is "binary input" in general, but simply illegal > input. The prompt expects, as it says, one of these 3 replies: > > CONFIRM DEPLOY_PROCEED=yes > CONFIRM DEPLOY_PROCEED=no > ABORT DEPLOY_PROCEED > > Instead, judging from the file supplied by Andrej, it gets from the engine: > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes > > So either the engine now sends, for some reason, 7169 null bytes, in > this response, or there is some low-level change causing this to be > eventually supplied to otopi - a change in apache-sshd, openssh, some > library, the kernel, no idea. > > Well, thinking a bit, I have a wild guess: Perhaps it's related to the > patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
> > > > > > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod > > method['method']() > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm > > prompt=True, > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm > > code=opcode, > > > > > >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: > >>> > >>> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock > >>> > >>> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote: > >>>> > >>>> > >>>> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: > >>>>> > >>>>> did you use mock to reproduce? > >>>> > >>>> > >>>> No, just run_suite under myself > >>>>> > >>>>> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). > >>>>>> > >>>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. > >>>>>> > >>>>>> Thanks > >>>>>> > >>>>>> Martin > >>>>>> > >>>>>> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: > >>>>>>> > >>>>>>> Full logs can be found here: > >>>>>>> > >>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... > >>>>>>> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> The previous regression was resolved and we now have a new regression. > >>>>>>>> > >>>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? > >>>>>>>> > >>>>>>>> The failure is add host on the upgrade suite. > >>>>>>>> > >>>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. > >>>>>>>> > >>>>>>>> error: > >>>>>>>> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) > >>>>>>>> (END) > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Dafna > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Martin Perina > >>>>>> Associate Manager, Software Engineering > >>>>>> Red Hat Czech s.r.o. > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Martin Perina > >>>> Associate Manager, Software Engineering > >>>> Red Hat Czech s.r.o. > >>> > >>> > >> > > > > > > > > -- > > Martin Perina > > Associate Manager, Software Engineering > > Red Hat Czech s.r.o. > > > > -- > Didi
-- Didi
-- Didi
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo@redhat.com
-- Didi
Didi, Is this what you are looking for https://ovirt-jira.atlassian.net/browse/OVIRT-2259 ? Galit On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com>
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori <
rnori@redhat.com> wrote:
> > I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes > > > > 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED > > ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes
Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies:
CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED
Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes
So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea.
Well, thinking a bit, I have a wild guess: Perhaps it's related to
wrote: the
patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
2018-09-14 15:49:23,032+0200 DEBUG otopi.context
context._executeMethod:143 method exception
Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm prompt=True, File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm code=opcode,
> > On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: >> >> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock >> >> >> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: >>> >>> >>> >>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: >>>> >>>> did you use mock to reproduce? >>> >>> >>> No, just run_suite under myself >>>> >>>> >>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: >>>>> >>>>> Hi, >>>>> >>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). >>>>> >>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >>>>> >>>>> Thanks >>>>> >>>>> Martin >>>>> >>>>> >>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: >>>>>> >>>>>> Full logs can be found here: >>>>>> >>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>>>>> >>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> The previous regression was resolved and we now have a new regression. >>>>>>> >>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >>>>>>> >>>>>>> The failure is add host on the upgrade suite. >>>>>>> >>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >>>>>>> >>>>>>> error: >>>>>>> >>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >>>>>>> (END) >>>>>>> >>>>>>> Thanks, >>>>>>> Dafna >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Martin Perina >>>>> Associate Manager, Software Engineering >>>>> Red Hat Czech s.r.o. >>>> >>>> >>> >>> >>> >>> -- >>> Martin Perina >>> Associate Manager, Software Engineering >>> Red Hat Czech s.r.o. >> >> >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Didi
-- Didi
-- Didi
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CG2IYPXSSEFTL6...
-- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
I'm pretty sure I verified this on el7 as well, i'll check again, but thinking about it, tar will stop when it gets to the first empty block, so if the record size on the engine's side is large and the end is filled with zeros, -b1 will make it stop at the first empty block so the next read on the host's side would get the trailing zeros which is what otopi reads. Btw, it could be a problem with deployed el7 systems as well, if for any reason the default on the host is set to something that is more than 20 blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on the host side). It's ok to revert the patch to fix the regression, but I don't see any other way other than -b1... perhaps add a `cat -` after to just read until EOF or something, or have otopi strip the input. On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth@redhat.com> wrote:
Didi,
Is this what you are looking for https://ovirt-jira.atlassian.net/browse/OVIRT-2259 ? Galit
On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/ job/ovirt-system-tests_manual/3189/
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com>
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com>
wrote:
On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com>
wrote:
> > > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < rnori@redhat.com> wrote: >> >> I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes >> >> >> >> 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine
>> >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes > > > Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input:
Not sure the issue is "binary input" in general, but simply illegal input. The prompt expects, as it says, one of these 3 replies:
CONFIRM DEPLOY_PROCEED=yes CONFIRM DEPLOY_PROCEED=no ABORT DEPLOY_PROCEED
Instead, judging from the file supplied by Andrej, it gets from the engine: <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes
So either the engine now sends, for some reason, 7169 null bytes, in this response, or there is some low-level change causing this to be eventually supplied to otopi - a change in apache-sshd, openssh, some library, the kernel, no idea.
Well, thinking a bit, I have a wild guess: Perhaps it's related to
patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context
context._executeMethod:143 method exception
> Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod > method['method']() > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py",
> prompt=True, > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py",
wrote: dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED the line 87, in _confirm line 478, in confirm
> code=opcode, > > >> >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: >>> >>> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock >>> >>> >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: >>>> >>>> >>>> >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: >>>>> >>>>> did you use mock to reproduce? >>>> >>>> >>>> No, just run_suite under myself >>>>> >>>>> >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). >>>>>> >>>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Martin >>>>>> >>>>>> >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: >>>>>>> >>>>>>> Full logs can be found here: >>>>>>> >>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/ job/ovirt-master_change-queue-tester/10307/artifact/upgrade- from-release-suite.el7.x86_64/test_logs/upgrade-from- release-suite-master/post-002_bootstrap.py/ >>>>>>> >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < dron@redhat.com> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> The previous regression was resolved and we now have a new regression. >>>>>>>> >>>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >>>>>>>> >>>>>>>> The failure is add host on the upgrade suite. >>>>>>>> >>>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >>>>>>>> >>>>>>>> error: >>>>>>>> >>>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >>>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >>>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core. vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core. vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >>>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >>>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >>>>>>>> (END) >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Dafna >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Martin Perina >>>>>> Associate Manager, Software Engineering >>>>>> Red Hat Czech s.r.o. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Martin Perina >>>> Associate Manager, Software Engineering >>>> Red Hat Czech s.r.o. >>> >>> >> > > > > -- > Martin Perina > Associate Manager, Software Engineering > Red Hat Czech s.r.o.
-- Didi
-- Didi
-- Didi
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/CG2IYPXSSEFTL6XCN72JHUSWOUY7QRSA/
--
GALIT ROSENTHAL
SOFTWARE ENGINEER
Red Hat
galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
host-deploy is still broken on master fc28 On Mon, Sep 17, 2018 at 8:01 AM, Yuval Turgeman <yturgema@redhat.com> wrote:
I'm pretty sure I verified this on el7 as well, i'll check again, but thinking about it, tar will stop when it gets to the first empty block, so if the record size on the engine's side is large and the end is filled with zeros, -b1 will make it stop at the first empty block so the next read on the host's side would get the trailing zeros which is what otopi reads. Btw, it could be a problem with deployed el7 systems as well, if for any reason the default on the host is set to something that is more than 20 blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on the host side). It's ok to revert the patch to fix the regression, but I don't see any other way other than -b1... perhaps add a `cat -` after to just read until EOF or something, or have otopi strip the input.
On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth@redhat.com> wrote:
Didi,
Is this what you are looking for https://ovirt-jira.atlassian.net/browse/OVIRT-2259 ? Galit
On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ ovirt-system-tests_manual/3189/
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com>
On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David <didi@redhat.com>
wrote:
> > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com> wrote: > > > > > > > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < rnori@redhat.com> wrote: > >> > >> I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes > >> > >> > >> > >> 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED > >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes > > > > > > Didi/Sandro, could you please take a look? Below error seems
> > Not sure the issue is "binary input" in general, but simply illegal > input. The prompt expects, as it says, one of these 3 replies: > > CONFIRM DEPLOY_PROCEED=yes > CONFIRM DEPLOY_PROCEED=no > ABORT DEPLOY_PROCEED > > Instead, judging from the file supplied by Andrej, it gets from
> <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes > > So either the engine now sends, for some reason, 7169 null bytes, in > this response, or there is some low-level change causing this to be > eventually supplied to otopi - a change in apache-sshd, openssh, some > library, the kernel, no idea. > > Well, thinking a bit, I have a wild guess: Perhaps it's related to
> patch introduced recently to change the tar blocking?
https://gerrit.ovirt.org/94357
I am leaving soon, perhaps someone can try the manual job with the result of the check-patch job for above patch, to see if it fixes. Otherwise I'll do this tomorrow.
> > > > > > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/otopi/context.py",
> > method['method']() > > File "/tmp/ovirt-O6CfS4aUHI/otopi-p lugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm > > prompt=True, > > File "/tmp/ovirt-O6CfS4aUHI/otopi-p lugins/otopi/dialog/machine.py", line 478, in confirm > > code=opcode, > > > > > >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: > >>> > >>> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock > >>> > >>> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: > >>>> > >>>> > >>>> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: > >>>>> > >>>>> did you use mock to reproduce? > >>>> > >>>> > >>>> No, just run_suite under myself > >>>>> > >>>>> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). > >>>>>> > >>>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. > >>>>>> > >>>>>> Thanks > >>>>>> > >>>>>> Martin > >>>>>> > >>>>>> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote: > >>>>>>> > >>>>>>> Full logs can be found here: > >>>>>>> > >>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ ovirt-master_change-queue-tester/10307/artifact/upgrade-from -release-suite.el7.x86_64/test_logs/upgrade-from-release- suite-master/post-002_bootstrap.py/ > >>>>>>> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < dron@redhat.com> wrote: > >>>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> The previous regression was resolved and we now have a new regression. > >>>>>>>> > >>>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? > >>>>>>>> > >>>>>>>> The failure is add host on the upgrade suite. > >>>>>>>> > >>>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. > >>>>>>>> > >>>>>>>> error: > >>>>>>>> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)";
wrote: like some issue in otopi, where an error is raised when handling binary input: the engine: the line 133, in _executeMethod trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True'
> >>>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) > >>>>>>>> (END) > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Dafna > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Martin Perina > >>>>>> Associate Manager, Software Engineering > >>>>>> Red Hat Czech s.r.o. > >>>>> > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Martin Perina > >>>> Associate Manager, Software Engineering > >>>> Red Hat Czech s.r.o. > >>> > >>> > >> > > > > > > > > -- > > Martin Perina > > Associate Manager, Software Engineering > > Red Hat Czech s.r.o. > > > > -- > Didi
-- Didi
-- Didi
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/communit y/about/community-guidelines/ List Archives: https://lists.ovirt.org/archiv es/list/infra@ovirt.org/message/CG2IYPXSSEFTL6XCN72JHUSWOUY7QRSA/
--
GALIT ROSENTHAL
SOFTWARE ENGINEER
Red Hat
galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/QMRM2INTCRDPT7GPF24EEPNJAZRP4CUQ/
On Mon, 17 Sep 2018, 16:25 Ravi Shankar Nori, <rnori@redhat.com> wrote:
host-deploy is still broken on master fc28
Yes, there are multiple issues on FC28, but the question is if this fixed OST on CentOS?
On Mon, Sep 17, 2018 at 8:01 AM, Yuval Turgeman <yturgema@redhat.com> wrote:
I'm pretty sure I verified this on el7 as well, i'll check again, but thinking about it, tar will stop when it gets to the first empty block, so if the record size on the engine's side is large and the end is filled with zeros, -b1 will make it stop at the first empty block so the next read on the host's side would get the trailing zeros which is what otopi reads. Btw, it could be a problem with deployed el7 systems as well, if for any reason the default on the host is set to something that is more than 20 blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on the host side). It's ok to revert the patch to fix the regression, but I don't see any other way other than -b1... perhaps add a `cat -` after to just read until EOF or something, or have otopi strip the input.
On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth@redhat.com> wrote:
Didi,
Is this what you are looking for https://ovirt-jira.atlassian.net/browse/OVIRT-2259 ? Galit
On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote:
Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David <didi@redhat.com>
> > On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David < didi@redhat.com> wrote: > > > > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina <mperina@redhat.com> wrote: > > > > > > > > > > > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < rnori@redhat.com> wrote: > > >> > > >> I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes > > >> > > >> > > >> > > >> 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED > > >> > > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes > > > > > > > > > Didi/Sandro, could you please take a look? Below error seems
> > > > Not sure the issue is "binary input" in general, but simply illegal > > input. The prompt expects, as it says, one of these 3 replies: > > > > CONFIRM DEPLOY_PROCEED=yes > > CONFIRM DEPLOY_PROCEED=no > > ABORT DEPLOY_PROCEED > > > > Instead, judging from the file supplied by Andrej, it gets from
> > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes > > > > So either the engine now sends, for some reason, 7169 null bytes, in > > this response, or there is some low-level change causing this to be > > eventually supplied to otopi - a change in apache-sshd, openssh, some > > library, the kernel, no idea. > > > > Well, thinking a bit, I have a wild guess: Perhaps it's related to the > > patch introduced recently to change the tar blocking? > > https://gerrit.ovirt.org/94357 > > I am leaving soon, perhaps someone can try the manual job with the > result of the check-patch job for above patch, to see if it fixes. > Otherwise I'll do this tomorrow. > > > > > > > > > > > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception > > > Traceback (most recent call last): > > > File "/usr/lib/python2.7/site-packages/otopi/context.py",
wrote: like some issue in otopi, where an error is raised when handling binary input: the engine: line 133, in _executeMethod
> > > method['method']() > > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm > > > prompt=True, > > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm > > > code=opcode, > > > > > > > > >> > > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: > > >>> > > >>> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock > > >>> > > >>> > > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: > > >>>> > > >>>> > > >>>> > > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: > > >>>>> > > >>>>> did you use mock to reproduce? > > >>>> > > >>>> > > >>>> No, just run_suite under myself > > >>>>> > > >>>>> > > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: > > >>>>>> > > >>>>>> Hi, > > >>>>>> > > >>>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). > > >>>>>> > > >>>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. > > >>>>>> > > >>>>>> Thanks > > >>>>>> > > >>>>>> Martin > > >>>>>> > > >>>>>> > > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron < dron@redhat.com> wrote: > > >>>>>>> > > >>>>>>> Full logs can be found here: > > >>>>>>> > > >>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... > > >>>>>>> > > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < dron@redhat.com> wrote: > > >>>>>>>> > > >>>>>>>> Hi, > > >>>>>>>> > > >>>>>>>> The previous regression was resolved and we now have a new regression. > > >>>>>>>> > > >>>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? > > >>>>>>>> > > >>>>>>>> The failure is add host on the upgrade suite. > > >>>>>>>> > > >>>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. > > >>>>>>>> > > >>>>>>>> error: > > >>>>>>>> > > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' > > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. > > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back > > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog > > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install > > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. > > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 > > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 > > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 > > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 > > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. > > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' > > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) > > >>>>>>>> (END) > > >>>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> Dafna > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Martin Perina > > >>>>>> Associate Manager, Software Engineering > > >>>>>> Red Hat Czech s.r.o. > > >>>>> > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Martin Perina > > >>>> Associate Manager, Software Engineering > > >>>> Red Hat Czech s.r.o. > > >>> > > >>> > > >> > > > > > > > > > > > > -- > > > Martin Perina > > > Associate Manager, Software Engineering > > > Red Hat Czech s.r.o. > > > > > > > > -- > > Didi > > > > -- > Didi
-- Didi
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CG2IYPXSSEFTL6...
--
GALIT ROSENTHAL
SOFTWARE ENGINEER
Red Hat
galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/QMRM2INTCRDPT7...
Ok, regarding the tar issue, there's another solution - since commons-compress hard coded the blocking factor to 1, while the default in tar is 20, we could continue creating the tar as we do today, but add a bunch of zeros to the end of the tarball. [yturgema@piggie ~/aa]$ ls -l ovirt-host-deploy_fc28.tar -rw-r--r--. 1 yturgema yturgema 2166272 Sep 2 17:03 ovirt-host-deploy_fc28.tar [yturgema@piggie ~/aa]$ python -c "print 10240-2166272%10240" 4608 [yturgema@piggie ~/aa]$ dd if=/dev/zero of=/dev/stdout bs=4608 count=1 | cat ovirt-host-deploy_fc28.tar - > new_host_deploy.tar 1+0 records in 1+0 records out 4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000373488 s, 12.3 MB/s [yturgema@piggie ~/aa]$ ls -l new_host_deploy.tar -rw-rw-r--. 1 yturgema yturgema 2170880 Sep 17 18:16 new_host_deploy.tar This would solve the problem, and not break el7, if this solution is acceptable, I can send a patch. On Mon, Sep 17, 2018 at 6:02 PM, Martin Perina <mperina@redhat.com> wrote:
On Mon, 17 Sep 2018, 16:25 Ravi Shankar Nori, <rnori@redhat.com> wrote:
host-deploy is still broken on master fc28
Yes, there are multiple issues on FC28, but the question is if this fixed OST on CentOS?
On Mon, Sep 17, 2018 at 8:01 AM, Yuval Turgeman <yturgema@redhat.com> wrote:
I'm pretty sure I verified this on el7 as well, i'll check again, but thinking about it, tar will stop when it gets to the first empty block, so if the record size on the engine's side is large and the end is filled with zeros, -b1 will make it stop at the first empty block so the next read on the host's side would get the trailing zeros which is what otopi reads. Btw, it could be a problem with deployed el7 systems as well, if for any reason the default on the host is set to something that is more than 20 blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on the host side). It's ok to revert the patch to fix the regression, but I don't see any other way other than -b1... perhaps add a `cat -` after to just read until EOF or something, or have otopi strip the input.
On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth@redhat.com> wrote:
Didi,
Is this what you are looking for https://ovirt-jira.atlassian.net/browse/OVIRT-2259 ? Galit
On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote: > > Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/ job/ovirt-system-tests_manual/3189/
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
> > > On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David < didi@redhat.com> wrote: >> >> On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David < didi@redhat.com> wrote: >> > >> > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina < mperina@redhat.com> wrote: >> > > >> > > >> > > >> > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < rnori@redhat.com> wrote: >> > >> >> > >> I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes >> > >> >> > >> >> > >> >> > >> 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED >> > >> >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes >> > > >> > > >> > > Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input: >> > >> > Not sure the issue is "binary input" in general, but simply illegal >> > input. The prompt expects, as it says, one of these 3 replies: >> > >> > CONFIRM DEPLOY_PROCEED=yes >> > CONFIRM DEPLOY_PROCEED=no >> > ABORT DEPLOY_PROCEED >> > >> > Instead, judging from the file supplied by Andrej, it gets from the engine: >> > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes >> > >> > So either the engine now sends, for some reason, 7169 null bytes, in >> > this response, or there is some low-level change causing this to be >> > eventually supplied to otopi - a change in apache-sshd, openssh, some >> > library, the kernel, no idea. >> > >> > Well, thinking a bit, I have a wild guess: Perhaps it's related to the >> > patch introduced recently to change the tar blocking? >> >> https://gerrit.ovirt.org/94357 >> >> I am leaving soon, perhaps someone can try the manual job with the >> result of the check-patch job for above patch, to see if it fixes. >> Otherwise I'll do this tomorrow. >> >> > >> > > >> > > >> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception >> > > Traceback (most recent call last): >> > > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >> > > method['method']() >> > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm >> > > prompt=True, >> > > File "/tmp/ovirt-O6CfS4aUHI/otopi- plugins/otopi/dialog/machine.py", line 478, in confirm >> > > code=opcode, >> > > >> > > >> > >> >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: >> > >>> >> > >>> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock >> > >>> >> > >>> >> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: >> > >>>> >> > >>>> >> > >>>> >> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: >> > >>>>> >> > >>>>> did you use mock to reproduce? >> > >>>> >> > >>>> >> > >>>> No, just run_suite under myself >> > >>>>> >> > >>>>> >> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: >> > >>>>>> >> > >>>>>> Hi, >> > >>>>>> >> > >>>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). >> > >>>>>> >> > >>>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >> > >>>>>> >> > >>>>>> Thanks >> > >>>>>> >> > >>>>>> Martin >> > >>>>>> >> > >>>>>> >> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron < dron@redhat.com> wrote: >> > >>>>>>> >> > >>>>>>> Full logs can be found here: >> > >>>>>>> >> > >>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/ job/ovirt-master_change-queue-tester/10307/artifact/upgrade- from-release-suite.el7.x86_64/test_logs/upgrade-from- release-suite-master/post-002_bootstrap.py/ >> > >>>>>>> >> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < dron@redhat.com> wrote: >> > >>>>>>>> >> > >>>>>>>> Hi, >> > >>>>>>>> >> > >>>>>>>> The previous regression was resolved and we now have a new regression. >> > >>>>>>>> >> > >>>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >> > >>>>>>>> >> > >>>>>>>> The failure is add host on the upgrade suite. >> > >>>>>>>> >> > >>>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >> > >>>>>>>> >> > >>>>>>>> error: >> > >>>>>>>> >> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >> > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >> > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >> > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >> > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >> > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core. vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >> > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core. vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >> > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >> > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >> > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >> > >>>>>>>> (END) >> > >>>>>>>> >> > >>>>>>>> Thanks, >> > >>>>>>>> Dafna >> > >>>>>>>> >> > >>>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> -- >> > >>>>>> Martin Perina >> > >>>>>> Associate Manager, Software Engineering >> > >>>>>> Red Hat Czech s.r.o. >> > >>>>> >> > >>>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> Martin Perina >> > >>>> Associate Manager, Software Engineering >> > >>>> Red Hat Czech s.r.o. >> > >>> >> > >>> >> > >> >> > > >> > > >> > > >> > > -- >> > > Martin Perina >> > > Associate Manager, Software Engineering >> > > Red Hat Czech s.r.o. >> > >> > >> > >> > -- >> > Didi >> >> >> >> -- >> Didi
-- Didi
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/ community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/CG2IYPXSSEFTL6XCN72JHUSWOUY7QRSA/
--
GALIT ROSENTHAL
SOFTWARE ENGINEER
Red Hat
galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/ message/QMRM2INTCRDPT7GPF24EEPNJAZRP4CUQ/
On Mon, Sep 17, 2018 at 6:35 PM Yuval Turgeman <yturgema@redhat.com> wrote:
Ok, regarding the tar issue, there's another solution - since commons-compress hard coded the blocking factor to 1, while the default in tar is 20, we could continue creating the tar as we do today, but add a bunch of zeros to the end of the tarball.
[yturgema@piggie ~/aa]$ ls -l ovirt-host-deploy_fc28.tar -rw-r--r--. 1 yturgema yturgema 2166272 Sep 2 17:03 ovirt-host-deploy_fc28.tar [yturgema@piggie ~/aa]$ python -c "print 10240-2166272%10240" 4608 [yturgema@piggie ~/aa]$ dd if=/dev/zero of=/dev/stdout bs=4608 count=1 | cat ovirt-host-deploy_fc28.tar - > new_host_deploy.tar 1+0 records in 1+0 records out 4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000373488 s, 12.3 MB/s [yturgema@piggie ~/aa]$ ls -l new_host_deploy.tar -rw-rw-r--. 1 yturgema yturgema 2170880 Sep 17 18:16 new_host_deploy.tar
This would solve the problem, and not break el7, if this solution is acceptable, I can send a patch.
This will probably work, but if you ask me, is too ugly. If we want to go this path, we probably have to find a reliable way to find out the blocking factor, or we risk another failure in the future. What about simply giving up on all of this and calling the external 'tar' utility also for creating the archive, instead of using Java?
On Mon, Sep 17, 2018 at 6:02 PM, Martin Perina <mperina@redhat.com> wrote:
On Mon, 17 Sep 2018, 16:25 Ravi Shankar Nori, <rnori@redhat.com> wrote:
host-deploy is still broken on master fc28
Yes, there are multiple issues on FC28, but the question is if this fixed OST on CentOS?
On Mon, Sep 17, 2018 at 8:01 AM, Yuval Turgeman <yturgema@redhat.com> wrote:
I'm pretty sure I verified this on el7 as well, i'll check again, but thinking about it, tar will stop when it gets to the first empty block, so if the record size on the engine's side is large and the end is filled with zeros, -b1 will make it stop at the first empty block so the next read on the host's side would get the trailing zeros which is what otopi reads. Btw, it could be a problem with deployed el7 systems as well, if for any reason the default on the host is set to something that is more than 20 blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on the host side). It's ok to revert the patch to fix the regression, but I don't see any other way other than -b1... perhaps add a `cat -` after to just read until EOF or something, or have otopi strip the input.
On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth@redhat.com> wrote:
Didi,
Is this what you are looking for https://ovirt-jira.atlassian.net/browse/OVIRT-2259 ? Galit
On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
> On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote: > > > > Didi, Marin, any update on the patch? > > Yes - it passed. Actually failed, but only after host-deploy: > > > https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... > > I'd rather not merge it as-is, because it will break fedora. > > If someone can have a look at the code generating the tar file, and > can see if > it's easy to make it work well for both centos and fedora, perhaps > by explicitly > setting all relevant params to some reasonable values, great. > Otherwise, I guess > we can merge for now, as fedora is still not supported anyway. > > Thanks, > > > > > > > On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David < > didi@redhat.com> wrote: > >> > >> On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David < > didi@redhat.com> wrote: > >> > > >> > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina < > mperina@redhat.com> wrote: > >> > > > >> > > > >> > > > >> > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < > rnori@redhat.com> wrote: > >> > >> > >> > >> I see the same errors on my dev env. From the logs attached > by Andrej the response received by otopi has a bunch of null chars before > the actual response CONFIRM DEPLOY_PROCEED=yes > >> > >> > >> > >> > >> > >> > >> > >> 2018-09-14 15:49:23,018+0200 DEBUG > otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND > ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED > >> > >> > >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes > >> > > > >> > > > >> > > Didi/Sandro, could you please take a look? Below error seems > like some issue in otopi, where an error is raised when handling binary > input: > >> > > >> > Not sure the issue is "binary input" in general, but simply > illegal > >> > input. The prompt expects, as it says, one of these 3 replies: > >> > > >> > CONFIRM DEPLOY_PROCEED=yes > >> > CONFIRM DEPLOY_PROCEED=no > >> > ABORT DEPLOY_PROCEED > >> > > >> > Instead, judging from the file supplied by Andrej, it gets from > the engine: > >> > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes > >> > > >> > So either the engine now sends, for some reason, 7169 null > bytes, in > >> > this response, or there is some low-level change causing this > to be > >> > eventually supplied to otopi - a change in apache-sshd, > openssh, some > >> > library, the kernel, no idea. > >> > > >> > Well, thinking a bit, I have a wild guess: Perhaps it's related > to the > >> > patch introduced recently to change the tar blocking? > >> > >> https://gerrit.ovirt.org/94357 > >> > >> I am leaving soon, perhaps someone can try the manual job with the > >> result of the check-patch job for above patch, to see if it fixes. > >> Otherwise I'll do this tomorrow. > >> > >> > > >> > > > >> > > > >> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context > context._executeMethod:143 method exception > >> > > Traceback (most recent call last): > >> > > File "/usr/lib/python2.7/site-packages/otopi/context.py", > line 133, in _executeMethod > >> > > method['method']() > >> > > File > "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line > 87, in _confirm > >> > > prompt=True, > >> > > File > "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in > confirm > >> > > code=opcode, > >> > > > >> > > > >> > >> > >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> > wrote: > >> > >>> > >> > >>> if you run it with mock you would remove any environmental > conditions that can effect the outcome so I recommend using mock > >> > >>> > >> > >>> > >> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < > mperina@redhat.com> wrote: > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron < > dron@redhat.com> wrote: > >> > >>>>> > >> > >>>>> did you use mock to reproduce? > >> > >>>> > >> > >>>> > >> > >>>> No, just run_suite under myself > >> > >>>>> > >> > >>>>> > >> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < > mperina@redhat.com> wrote: > >> > >>>>>> > >> > >>>>>> Hi, > >> > >>>>>> > >> > >>>>>> the problem is that we haven't fetched the temporary > host-deploy log from /tmp directory, so we don't know which string that > host-deploy process sent to engine is causing that issue. I tried to > reproduce on my local machine, but I was unable to reproduce it, > 002_bootstrap phase finished successfully (other phases are still running). > >> > >>>>>> > >> > >>>>>> So if anyone is able to reproduce, please try to fetch > host-deploy log from /tmp directory after the error is raised and share it. > >> > >>>>>> > >> > >>>>>> Thanks > >> > >>>>>> > >> > >>>>>> Martin > >> > >>>>>> > >> > >>>>>> > >> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron < > dron@redhat.com> wrote: > >> > >>>>>>> > >> > >>>>>>> Full logs can be found here: > >> > >>>>>>> > >> > >>>>>>> > https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... > >> > >>>>>>> > >> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < > dron@redhat.com> wrote: > >> > >>>>>>>> > >> > >>>>>>>> Hi, > >> > >>>>>>>> > >> > >>>>>>>> The previous regression was resolved and we now have a > new regression. > >> > >>>>>>>> > >> > >>>>>>>> I don't think that the reported change is related so > can someone from ovirt-engine take a look? > >> > >>>>>>>> > >> > >>>>>>>> The failure is add host on the upgrade suite. > >> > >>>>>>>> > >> > >>>>>>>> Please note that we have not had an engine-ovirt build > for over 10 days due to several consecutive regressions and I would ask you > to stop merging until we can stabilize the project and have a new package > of engine. > >> > >>>>>>>> > >> > >>>>>>>> error: > >> > >>>>>>>> > >> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO > [org.ovirt.engine.core.uutils.ssh.SSHDialog] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute > 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; > MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap > "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && > "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine > DIALOG/customization=bool:True' > >> > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing > Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. > >> > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO > [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) > [5c91fcbd] transaction rolled back > >> > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR > [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] > Error during deploy dialog > >> > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR > [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host > lago-upgrade-from-release-suite-master-host-0 install > >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: > VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during > installation of Host lago-upgrade-from-release-suite-master-host-0: > CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested > exception is org.postgresql.util.PSQLException: ERROR: invalid byte > sequence for encoding "UTF8": 0x00. > >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR > [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host > lago-upgrade-from-release-suite-master-host-0 install, preferring first > exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; > nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte > sequence for encoding "UTF8": 0x00 > >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation > failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', > 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; > SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte > sequence for encoding "UTF8": 0x00; nested exception is > org.postgresql.util.PSQLException: ERROR: invalid byte sequence for > encoding "UTF8": 0x00 > >> > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO > [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, > SetVdsStatusVDSCommand(HostName = > lago-upgrade-from-release-suite-master-host-0, > SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', > status='InstallFailed', nonOperationalReason='NONE', > stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 > >> > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO > [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, > SetVdsStatusVDSCommand, return: , log id: 146cdc08 > >> > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: > VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 > installation failed. CallableStatementCallback; SQL [{call > insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for > encoding "UTF8": 0x00; nested exception is > org.postgresql.util.PSQLException: ERROR: invalid byte sequence for > encoding "UTF8": 0x00. > >> > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO > [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object > 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', > sharedLocks=''}' > >> > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO > [org.ovirt.engine.core.bll.quota.QuotaManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. > (19 msec) > >> > >>>>>>>> (END) > >> > >>>>>>>> > >> > >>>>>>>> Thanks, > >> > >>>>>>>> Dafna > >> > >>>>>>>> > >> > >>>>>>> > >> > >>>>>> > >> > >>>>>> > >> > >>>>>> > >> > >>>>>> -- > >> > >>>>>> Martin Perina > >> > >>>>>> Associate Manager, Software Engineering > >> > >>>>>> Red Hat Czech s.r.o. > >> > >>>>> > >> > >>>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> -- > >> > >>>> Martin Perina > >> > >>>> Associate Manager, Software Engineering > >> > >>>> Red Hat Czech s.r.o. > >> > >>> > >> > >>> > >> > >> > >> > > > >> > > > >> > > > >> > > -- > >> > > Martin Perina > >> > > Associate Manager, Software Engineering > >> > > Red Hat Czech s.r.o. > >> > > >> > > >> > > >> > -- > >> > Didi > >> > >> > >> > >> -- > >> Didi > > > > -- > Didi > _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CG2IYPXSSEFTL6...
--
GALIT ROSENTHAL
SOFTWARE ENGINEER
Red Hat
galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/QMRM2INTCRDPT7...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7MG4ZPPXOVMIET...
-- Didi
That would work too... the end result would be same, tar pads the last record with zeros as well On Thu, Sep 20, 2018 at 9:51 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 6:35 PM Yuval Turgeman <yturgema@redhat.com> wrote:
Ok, regarding the tar issue, there's another solution - since commons-compress hard coded the blocking factor to 1, while the default in tar is 20, we could continue creating the tar as we do today, but add a bunch of zeros to the end of the tarball.
[yturgema@piggie ~/aa]$ ls -l ovirt-host-deploy_fc28.tar -rw-r--r--. 1 yturgema yturgema 2166272 Sep 2 17:03 ovirt-host-deploy_fc28.tar [yturgema@piggie ~/aa]$ python -c "print 10240-2166272%10240" 4608 [yturgema@piggie ~/aa]$ dd if=/dev/zero of=/dev/stdout bs=4608 count=1 | cat ovirt-host-deploy_fc28.tar - > new_host_deploy.tar 1+0 records in 1+0 records out 4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000373488 s, 12.3 MB/s [yturgema@piggie ~/aa]$ ls -l new_host_deploy.tar -rw-rw-r--. 1 yturgema yturgema 2170880 Sep 17 18:16 new_host_deploy.tar
This would solve the problem, and not break el7, if this solution is acceptable, I can send a patch.
This will probably work, but if you ask me, is too ugly. If we want to go this path, we probably have to find a reliable way to find out the blocking factor, or we risk another failure in the future.
What about simply giving up on all of this and calling the external 'tar' utility also for creating the archive, instead of using Java?
On Mon, Sep 17, 2018 at 6:02 PM, Martin Perina <mperina@redhat.com> wrote:
On Mon, 17 Sep 2018, 16:25 Ravi Shankar Nori, <rnori@redhat.com> wrote:
host-deploy is still broken on master fc28
Yes, there are multiple issues on FC28, but the question is if this fixed OST on CentOS?
On Mon, Sep 17, 2018 at 8:01 AM, Yuval Turgeman <yturgema@redhat.com> wrote:
I'm pretty sure I verified this on el7 as well, i'll check again, but thinking about it, tar will stop when it gets to the first empty block, so if the record size on the engine's side is large and the end is filled with zeros, -b1 will make it stop at the first empty block so the next read on the host's side would get the trailing zeros which is what otopi reads. Btw, it could be a problem with deployed el7 systems as well, if for any reason the default on the host is set to something that is more than 20 blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on the host side). It's ok to revert the patch to fix the regression, but I don't see any other way other than -b1... perhaps add a `cat -` after to just read until EOF or something, or have otopi strip the input.
On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth@redhat.com> wrote:
Didi,
Is this what you are looking for https://ovirt-jira.atlassian.net/browse/OVIRT-2259 ? Galit
On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote:
> I think that in ovirt-engine we currently only build to centos. > since we have not had an engine build for 2 weeks (on master) I > think we should merge and worry about fc28 once it would be relevant. > > the failure we have now could be another regression missed since the > project has been broken for two weeks. > > Thanks, > Dafna > > > > On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> > wrote: > >> On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote: >> > >> > Didi, Marin, any update on the patch? >> >> Yes - it passed. Actually failed, but only after host-deploy: >> >> >> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... >> >> I'd rather not merge it as-is, because it will break fedora. >> >> If someone can have a look at the code generating the tar file, and >> can see if >> it's easy to make it work well for both centos and fedora, perhaps >> by explicitly >> setting all relevant params to some reasonable values, great. >> Otherwise, I guess >> we can merge for now, as fedora is still not supported anyway. >> >> Thanks, >> >> > >> > >> > On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David < >> didi@redhat.com> wrote: >> >> >> >> On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David < >> didi@redhat.com> wrote: >> >> > >> >> > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina < >> mperina@redhat.com> wrote: >> >> > > >> >> > > >> >> > > >> >> > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < >> rnori@redhat.com> wrote: >> >> > >> >> >> > >> I see the same errors on my dev env. From the logs attached >> by Andrej the response received by otopi has a bunch of null chars before >> the actual response CONFIRM DEPLOY_PROCEED=yes >> >> > >> >> >> > >> >> >> > >> >> >> > >> 2018-09-14 15:49:23,018+0200 DEBUG >> otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND >> ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED >> >> > >> >> >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes >> >> > > >> >> > > >> >> > > Didi/Sandro, could you please take a look? Below error seems >> like some issue in otopi, where an error is raised when handling binary >> input: >> >> > >> >> > Not sure the issue is "binary input" in general, but simply >> illegal >> >> > input. The prompt expects, as it says, one of these 3 replies: >> >> > >> >> > CONFIRM DEPLOY_PROCEED=yes >> >> > CONFIRM DEPLOY_PROCEED=no >> >> > ABORT DEPLOY_PROCEED >> >> > >> >> > Instead, judging from the file supplied by Andrej, it gets >> from the engine: >> >> > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes >> >> > >> >> > So either the engine now sends, for some reason, 7169 null >> bytes, in >> >> > this response, or there is some low-level change causing this >> to be >> >> > eventually supplied to otopi - a change in apache-sshd, >> openssh, some >> >> > library, the kernel, no idea. >> >> > >> >> > Well, thinking a bit, I have a wild guess: Perhaps it's >> related to the >> >> > patch introduced recently to change the tar blocking? >> >> >> >> https://gerrit.ovirt.org/94357 >> >> >> >> I am leaving soon, perhaps someone can try the manual job with >> the >> >> result of the check-patch job for above patch, to see if it >> fixes. >> >> Otherwise I'll do this tomorrow. >> >> >> >> > >> >> > > >> >> > > >> >> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context >> context._executeMethod:143 method exception >> >> > > Traceback (most recent call last): >> >> > > File "/usr/lib/python2.7/site-packages/otopi/context.py", >> line 133, in _executeMethod >> >> > > method['method']() >> >> > > File >> "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line >> 87, in _confirm >> >> > > prompt=True, >> >> > > File >> "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in >> confirm >> >> > > code=opcode, >> >> > > >> >> > > >> >> > >> >> >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron < >> dron@redhat.com> wrote: >> >> > >>> >> >> > >>> if you run it with mock you would remove any environmental >> conditions that can effect the outcome so I recommend using mock >> >> > >>> >> >> > >>> >> >> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < >> mperina@redhat.com> wrote: >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron < >> dron@redhat.com> wrote: >> >> > >>>>> >> >> > >>>>> did you use mock to reproduce? >> >> > >>>> >> >> > >>>> >> >> > >>>> No, just run_suite under myself >> >> > >>>>> >> >> > >>>>> >> >> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < >> mperina@redhat.com> wrote: >> >> > >>>>>> >> >> > >>>>>> Hi, >> >> > >>>>>> >> >> > >>>>>> the problem is that we haven't fetched the temporary >> host-deploy log from /tmp directory, so we don't know which string that >> host-deploy process sent to engine is causing that issue. I tried to >> reproduce on my local machine, but I was unable to reproduce it, >> 002_bootstrap phase finished successfully (other phases are still running). >> >> > >>>>>> >> >> > >>>>>> So if anyone is able to reproduce, please try to fetch >> host-deploy log from /tmp directory after the error is raised and share it. >> >> > >>>>>> >> >> > >>>>>> Thanks >> >> > >>>>>> >> >> > >>>>>> Martin >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron < >> dron@redhat.com> wrote: >> >> > >>>>>>> >> >> > >>>>>>> Full logs can be found here: >> >> > >>>>>>> >> >> > >>>>>>> >> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >> >> > >>>>>>> >> >> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < >> dron@redhat.com> wrote: >> >> > >>>>>>>> >> >> > >>>>>>>> Hi, >> >> > >>>>>>>> >> >> > >>>>>>>> The previous regression was resolved and we now have >> a new regression. >> >> > >>>>>>>> >> >> > >>>>>>>> I don't think that the reported change is related so >> can someone from ovirt-engine take a look? >> >> > >>>>>>>> >> >> > >>>>>>>> The failure is add host on the upgrade suite. >> >> > >>>>>>>> >> >> > >>>>>>>> Please note that we have not had an engine-ovirt >> build for over 10 days due to several consecutive regressions and I would >> ask you to stop merging until we can stabilize the project and have a new >> package of engine. >> >> > >>>>>>>> >> >> > >>>>>>>> error: >> >> > >>>>>>>> >> >> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO >> [org.ovirt.engine.core.uutils.ssh.SSHDialog] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute >> 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; >> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap >> "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > >> /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && >> "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >> DIALOG/customization=bool:True' >> >> > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing >> Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >> >> > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) >> [5c91fcbd] transaction rolled back >> >> > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] >> Error during deploy dialog >> >> > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >> lago-upgrade-from-release-suite-master-host-0 install >> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >> VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during >> installation of Host lago-upgrade-from-release-suite-master-host-0: >> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >> exception is org.postgresql.util.PSQLException: ERROR: invalid byte >> sequence for encoding "UTF8": 0x00. >> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >> lago-upgrade-from-release-suite-master-host-0 install, preferring first >> exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; >> nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte >> sequence for encoding "UTF8": 0x00 >> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation >> failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', >> 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; >> SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte >> sequence for encoding "UTF8": 0x00; nested exception is >> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >> encoding "UTF8": 0x00 >> >> > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO >> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, >> SetVdsStatusVDSCommand(HostName = >> lago-upgrade-from-release-suite-master-host-0, >> SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', >> status='InstallFailed', nonOperationalReason='NONE', >> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >> >> > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO >> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, >> SetVdsStatusVDSCommand, return: , log id: 146cdc08 >> >> > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >> VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 >> installation failed. CallableStatementCallback; SQL [{call >> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >> encoding "UTF8": 0x00; nested exception is >> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >> encoding "UTF8": 0x00. >> >> > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO >> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object >> 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', >> sharedLocks=''}' >> >> > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO >> [org.ovirt.engine.core.bll.quota.QuotaManager] >> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. >> (19 msec) >> >> > >>>>>>>> (END) >> >> > >>>>>>>> >> >> > >>>>>>>> Thanks, >> >> > >>>>>>>> Dafna >> >> > >>>>>>>> >> >> > >>>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> -- >> >> > >>>>>> Martin Perina >> >> > >>>>>> Associate Manager, Software Engineering >> >> > >>>>>> Red Hat Czech s.r.o. >> >> > >>>>> >> >> > >>>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> -- >> >> > >>>> Martin Perina >> >> > >>>> Associate Manager, Software Engineering >> >> > >>>> Red Hat Czech s.r.o. >> >> > >>> >> >> > >>> >> >> > >> >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > Martin Perina >> >> > > Associate Manager, Software Engineering >> >> > > Red Hat Czech s.r.o. >> >> > >> >> > >> >> > >> >> > -- >> >> > Didi >> >> >> >> >> >> >> >> -- >> >> Didi >> >> >> >> -- >> Didi >> > _______________________________________________ > Infra mailing list -- infra@ovirt.org > To unsubscribe send an email to infra-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CG2IYPXSSEFTL6... >
--
GALIT ROSENTHAL
SOFTWARE ENGINEER
Red Hat
galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/QMRM2INTCRDPT7...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7MG4ZPPXOVMIET...
-- Didi
Hi, I hit this problem again. Otopi sends many zero bytes before a response. This time, on an engine installed from master RPM, not dev environment. The engine and host machines are latest Centos 7. Is there some workaround for deploying the host? Thanks, Andrej On Thu, 20 Sep 2018 at 08:59, Yuval Turgeman <yturgema@redhat.com> wrote:
That would work too... the end result would be same, tar pads the last record with zeros as well
On Thu, Sep 20, 2018 at 9:51 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 6:35 PM Yuval Turgeman <yturgema@redhat.com> wrote:
Ok, regarding the tar issue, there's another solution - since commons-compress hard coded the blocking factor to 1, while the default in tar is 20, we could continue creating the tar as we do today, but add a bunch of zeros to the end of the tarball.
[yturgema@piggie ~/aa]$ ls -l ovirt-host-deploy_fc28.tar -rw-r--r--. 1 yturgema yturgema 2166272 Sep 2 17:03 ovirt-host-deploy_fc28.tar [yturgema@piggie ~/aa]$ python -c "print 10240-2166272%10240" 4608 [yturgema@piggie ~/aa]$ dd if=/dev/zero of=/dev/stdout bs=4608 count=1 | cat ovirt-host-deploy_fc28.tar - > new_host_deploy.tar 1+0 records in 1+0 records out 4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000373488 s, 12.3 MB/s [yturgema@piggie ~/aa]$ ls -l new_host_deploy.tar -rw-rw-r--. 1 yturgema yturgema 2170880 Sep 17 18:16 new_host_deploy.tar
This would solve the problem, and not break el7, if this solution is acceptable, I can send a patch.
This will probably work, but if you ask me, is too ugly. If we want to go this path, we probably have to find a reliable way to find out the blocking factor, or we risk another failure in the future.
What about simply giving up on all of this and calling the external 'tar' utility also for creating the archive, instead of using Java?
On Mon, Sep 17, 2018 at 6:02 PM, Martin Perina <mperina@redhat.com> wrote:
On Mon, 17 Sep 2018, 16:25 Ravi Shankar Nori, <rnori@redhat.com> wrote:
host-deploy is still broken on master fc28
Yes, there are multiple issues on FC28, but the question is if this fixed OST on CentOS?
On Mon, Sep 17, 2018 at 8:01 AM, Yuval Turgeman <yturgema@redhat.com> wrote:
I'm pretty sure I verified this on el7 as well, i'll check again, but thinking about it, tar will stop when it gets to the first empty block, so if the record size on the engine's side is large and the end is filled with zeros, -b1 will make it stop at the first empty block so the next read on the host's side would get the trailing zeros which is what otopi reads. Btw, it could be a problem with deployed el7 systems as well, if for any reason the default on the host is set to something that is more than 20 blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on the host side). It's ok to revert the patch to fix the regression, but I don't see any other way other than -b1... perhaps add a `cat -` after to just read until EOF or something, or have otopi strip the input.
On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth@redhat.com > wrote:
> Didi, > > Is this what you are looking for > https://ovirt-jira.atlassian.net/browse/OVIRT-2259 > ? > Galit > > On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote: > >> I think that in ovirt-engine we currently only build to centos. >> since we have not had an engine build for 2 weeks (on master) I >> think we should merge and worry about fc28 once it would be relevant. >> >> the failure we have now could be another regression missed since >> the project has been broken for two weeks. >> >> Thanks, >> Dafna >> >> >> >> On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David < >> didi@redhat.com> wrote: >> >>> On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> >>> wrote: >>> > >>> > Didi, Marin, any update on the patch? >>> >>> Yes - it passed. Actually failed, but only after host-deploy: >>> >>> >>> https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests... >>> >>> I'd rather not merge it as-is, because it will break fedora. >>> >>> If someone can have a look at the code generating the tar file, >>> and can see if >>> it's easy to make it work well for both centos and fedora, perhaps >>> by explicitly >>> setting all relevant params to some reasonable values, great. >>> Otherwise, I guess >>> we can merge for now, as fedora is still not supported anyway. >>> >>> Thanks, >>> >>> > >>> > >>> > On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David < >>> didi@redhat.com> wrote: >>> >> >>> >> On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David < >>> didi@redhat.com> wrote: >>> >> > >>> >> > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina < >>> mperina@redhat.com> wrote: >>> >> > > >>> >> > > >>> >> > > >>> >> > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < >>> rnori@redhat.com> wrote: >>> >> > >> >>> >> > >> I see the same errors on my dev env. From the logs >>> attached by Andrej the response received by otopi has a bunch of null chars >>> before the actual response CONFIRM DEPLOY_PROCEED=yes >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > >> 2018-09-14 15:49:23,018+0200 DEBUG >>> otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND >>> ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED >>> >> > >> >>> >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes >>> >> > > >>> >> > > >>> >> > > Didi/Sandro, could you please take a look? Below error >>> seems like some issue in otopi, where an error is raised when handling >>> binary input: >>> >> > >>> >> > Not sure the issue is "binary input" in general, but simply >>> illegal >>> >> > input. The prompt expects, as it says, one of these 3 replies: >>> >> > >>> >> > CONFIRM DEPLOY_PROCEED=yes >>> >> > CONFIRM DEPLOY_PROCEED=no >>> >> > ABORT DEPLOY_PROCEED >>> >> > >>> >> > Instead, judging from the file supplied by Andrej, it gets >>> from the engine: >>> >> > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes >>> >> > >>> >> > So either the engine now sends, for some reason, 7169 null >>> bytes, in >>> >> > this response, or there is some low-level change causing this >>> to be >>> >> > eventually supplied to otopi - a change in apache-sshd, >>> openssh, some >>> >> > library, the kernel, no idea. >>> >> > >>> >> > Well, thinking a bit, I have a wild guess: Perhaps it's >>> related to the >>> >> > patch introduced recently to change the tar blocking? >>> >> >>> >> https://gerrit.ovirt.org/94357 >>> >> >>> >> I am leaving soon, perhaps someone can try the manual job with >>> the >>> >> result of the check-patch job for above patch, to see if it >>> fixes. >>> >> Otherwise I'll do this tomorrow. >>> >> >>> >> > >>> >> > > >>> >> > > >>> >> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context >>> context._executeMethod:143 method exception >>> >> > > Traceback (most recent call last): >>> >> > > File "/usr/lib/python2.7/site-packages/otopi/context.py", >>> line 133, in _executeMethod >>> >> > > method['method']() >>> >> > > File >>> "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line >>> 87, in _confirm >>> >> > > prompt=True, >>> >> > > File >>> "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in >>> confirm >>> >> > > code=opcode, >>> >> > > >>> >> > > >>> >> > >> >>> >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron < >>> dron@redhat.com> wrote: >>> >> > >>> >>> >> > >>> if you run it with mock you would remove any >>> environmental conditions that can effect the outcome so I recommend using >>> mock >>> >> > >>> >>> >> > >>> >>> >> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < >>> mperina@redhat.com> wrote: >>> >> > >>>> >>> >> > >>>> >>> >> > >>>> >>> >> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron < >>> dron@redhat.com> wrote: >>> >> > >>>>> >>> >> > >>>>> did you use mock to reproduce? >>> >> > >>>> >>> >> > >>>> >>> >> > >>>> No, just run_suite under myself >>> >> > >>>>> >>> >> > >>>>> >>> >> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < >>> mperina@redhat.com> wrote: >>> >> > >>>>>> >>> >> > >>>>>> Hi, >>> >> > >>>>>> >>> >> > >>>>>> the problem is that we haven't fetched the temporary >>> host-deploy log from /tmp directory, so we don't know which string that >>> host-deploy process sent to engine is causing that issue. I tried to >>> reproduce on my local machine, but I was unable to reproduce it, >>> 002_bootstrap phase finished successfully (other phases are still running). >>> >> > >>>>>> >>> >> > >>>>>> So if anyone is able to reproduce, please try to fetch >>> host-deploy log from /tmp directory after the error is raised and share it. >>> >> > >>>>>> >>> >> > >>>>>> Thanks >>> >> > >>>>>> >>> >> > >>>>>> Martin >>> >> > >>>>>> >>> >> > >>>>>> >>> >> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron < >>> dron@redhat.com> wrote: >>> >> > >>>>>>> >>> >> > >>>>>>> Full logs can be found here: >>> >> > >>>>>>> >>> >> > >>>>>>> >>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>> >> > >>>>>>> >>> >> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < >>> dron@redhat.com> wrote: >>> >> > >>>>>>>> >>> >> > >>>>>>>> Hi, >>> >> > >>>>>>>> >>> >> > >>>>>>>> The previous regression was resolved and we now have >>> a new regression. >>> >> > >>>>>>>> >>> >> > >>>>>>>> I don't think that the reported change is related so >>> can someone from ovirt-engine take a look? >>> >> > >>>>>>>> >>> >> > >>>>>>>> The failure is add host on the upgrade suite. >>> >> > >>>>>>>> >>> >> > >>>>>>>> Please note that we have not had an engine-ovirt >>> build for over 10 days due to several consecutive regressions and I would >>> ask you to stop merging until we can stabilize the project and have a new >>> package of engine. >>> >> > >>>>>>>> >>> >> > >>>>>>>> error: >>> >> > >>>>>>>> >>> >> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO >>> [org.ovirt.engine.core.uutils.ssh.SSHDialog] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute >>> 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; >>> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap >>> "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > >>> /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && >>> "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >>> DIALOG/customization=bool:True' >>> >> > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing >>> Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >>> >> > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO >>> [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) >>> [5c91fcbd] transaction rolled back >>> >> > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR >>> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] >>> Error during deploy dialog >>> >> > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR >>> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >>> lago-upgrade-from-release-suite-master-host-0 install >>> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >>> VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during >>> installation of Host lago-upgrade-from-release-suite-master-host-0: >>> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >>> exception is org.postgresql.util.PSQLException: ERROR: invalid byte >>> sequence for encoding "UTF8": 0x00. >>> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >>> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >>> lago-upgrade-from-release-suite-master-host-0 install, preferring first >>> exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, >>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>> ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; >>> nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte >>> sequence for encoding "UTF8": 0x00 >>> >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR >>> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation >>> failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', >>> 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; >>> SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte >>> sequence for encoding "UTF8": 0x00; nested exception is >>> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >>> encoding "UTF8": 0x00 >>> >> > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, >>> SetVdsStatusVDSCommand(HostName = >>> lago-upgrade-from-release-suite-master-host-0, >>> SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', >>> status='InstallFailed', nonOperationalReason='NONE', >>> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >>> >> > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, >>> SetVdsStatusVDSCommand, return: , log id: 146cdc08 >>> >> > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >>> VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 >>> installation failed. CallableStatementCallback; SQL [{call >>> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >>> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >>> encoding "UTF8": 0x00; nested exception is >>> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >>> encoding "UTF8": 0x00. >>> >> > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO >>> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >>> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object >>> 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', >>> sharedLocks=''}' >>> >> > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO >>> [org.ovirt.engine.core.bll.quota.QuotaManager] >>> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. >>> (19 msec) >>> >> > >>>>>>>> (END) >>> >> > >>>>>>>> >>> >> > >>>>>>>> Thanks, >>> >> > >>>>>>>> Dafna >>> >> > >>>>>>>> >>> >> > >>>>>>> >>> >> > >>>>>> >>> >> > >>>>>> >>> >> > >>>>>> >>> >> > >>>>>> -- >>> >> > >>>>>> Martin Perina >>> >> > >>>>>> Associate Manager, Software Engineering >>> >> > >>>>>> Red Hat Czech s.r.o. >>> >> > >>>>> >>> >> > >>>>> >>> >> > >>>> >>> >> > >>>> >>> >> > >>>> >>> >> > >>>> -- >>> >> > >>>> Martin Perina >>> >> > >>>> Associate Manager, Software Engineering >>> >> > >>>> Red Hat Czech s.r.o. >>> >> > >>> >>> >> > >>> >>> >> > >> >>> >> > > >>> >> > > >>> >> > > >>> >> > > -- >>> >> > > Martin Perina >>> >> > > Associate Manager, Software Engineering >>> >> > > Red Hat Czech s.r.o. >>> >> > >>> >> > >>> >> > >>> >> > -- >>> >> > Didi >>> >> >>> >> >>> >> >>> >> -- >>> >> Didi >>> >>> >>> >>> -- >>> Didi >>> >> _______________________________________________ >> Infra mailing list -- infra@ovirt.org >> To unsubscribe send an email to infra-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CG2IYPXSSEFTL6... >> > > > -- > > GALIT ROSENTHAL > > SOFTWARE ENGINEER > > Red Hat > > <https://www.redhat.com/> > > galit@gmail.com T: 972-9-7692230 > <https://red.ht/sig> >
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/QMRM2INTCRDPT7...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/7MG4ZPPXOVMIET...
-- Didi
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/RBXX64O6WXL2VF...
I had to re-trigger didi's fix due to a issue in Jenkins but I can now confirm that the issue is resolved in ovirt-engine and that we have a new ovirt-engine package in tested for el7: https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... https://resources.ovirt.org/repos/ovirt/tested/master/rpm/el7/noarch/ On Mon, Sep 17, 2018 at 4:04 PM Martin Perina <mperina@redhat.com> wrote:
On Mon, 17 Sep 2018, 16:25 Ravi Shankar Nori, <rnori@redhat.com> wrote:
host-deploy is still broken on master fc28
Yes, there are multiple issues on FC28, but the question is if this fixed OST on CentOS?
On Mon, Sep 17, 2018 at 8:01 AM, Yuval Turgeman <yturgema@redhat.com> wrote:
I'm pretty sure I verified this on el7 as well, i'll check again, but thinking about it, tar will stop when it gets to the first empty block, so if the record size on the engine's side is large and the end is filled with zeros, -b1 will make it stop at the first empty block so the next read on the host's side would get the trailing zeros which is what otopi reads. Btw, it could be a problem with deployed el7 systems as well, if for any reason the default on the host is set to something that is more than 20 blocks (can be set with export TAR_BLOCKING_FACTOR for the root account on the host side). It's ok to revert the patch to fix the regression, but I don't see any other way other than -b1... perhaps add a `cat -` after to just read until EOF or something, or have otopi strip the input.
On Mon, Sep 17, 2018 at 2:30 PM, Galit Rosenthal <grosenth@redhat.com> wrote:
Didi,
Is this what you are looking for https://ovirt-jira.atlassian.net/browse/OVIRT-2259 ? Galit
On Mon, Sep 17, 2018 at 1:54 PM Dafna Ron <dron@redhat.com> wrote:
I think that in ovirt-engine we currently only build to centos. since we have not had an engine build for 2 weeks (on master) I think we should merge and worry about fc28 once it would be relevant.
the failure we have now could be another regression missed since the project has been broken for two weeks.
Thanks, Dafna
On Mon, Sep 17, 2018 at 10:30 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Sep 17, 2018 at 11:49 AM Dafna Ron <dron@redhat.com> wrote: > > Didi, Marin, any update on the patch?
Yes - it passed. Actually failed, but only after host-deploy:
https://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests...
I'd rather not merge it as-is, because it will break fedora.
If someone can have a look at the code generating the tar file, and can see if it's easy to make it work well for both centos and fedora, perhaps by explicitly setting all relevant params to some reasonable values, great. Otherwise, I guess we can merge for now, as fedora is still not supported anyway.
Thanks,
> > > On Sun, Sep 16, 2018 at 11:09 AM Yedidyah Bar David < didi@redhat.com> wrote: >> >> On Sun, Sep 16, 2018 at 12:53 PM Yedidyah Bar David < didi@redhat.com> wrote: >> > >> > On Fri, Sep 14, 2018 at 6:06 PM Martin Perina < mperina@redhat.com> wrote: >> > > >> > > >> > > >> > > On Fri, Sep 14, 2018 at 4:51 PM, Ravi Shankar Nori < rnori@redhat.com> wrote: >> > >> >> > >> I see the same errors on my dev env. From the logs attached by Andrej the response received by otopi has a bunch of null chars before the actual response CONFIRM DEPLOY_PROCEED=yes >> > >> >> > >> >> > >> >> > >> 2018-09-14 15:49:23,018+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND ### Response is CONFIRM DEPLOY_PROCEED=yes|no or ABORT DEPLOY_PROCEED >> > >> >> > >> ^@^@^@^@^@^@^@^@^@CONFIRM DEPLOY_PROCEED=yes >> > > >> > > >> > > Didi/Sandro, could you please take a look? Below error seems like some issue in otopi, where an error is raised when handling binary input: >> > >> > Not sure the issue is "binary input" in general, but simply illegal >> > input. The prompt expects, as it says, one of these 3 replies: >> > >> > CONFIRM DEPLOY_PROCEED=yes >> > CONFIRM DEPLOY_PROCEED=no >> > ABORT DEPLOY_PROCEED >> > >> > Instead, judging from the file supplied by Andrej, it gets from the engine: >> > <7169 null bytes>CONFIRM DEPLOY_PROCEED=yes >> > >> > So either the engine now sends, for some reason, 7169 null bytes, in >> > this response, or there is some low-level change causing this to be >> > eventually supplied to otopi - a change in apache-sshd, openssh, some >> > library, the kernel, no idea. >> > >> > Well, thinking a bit, I have a wild guess: Perhaps it's related to the >> > patch introduced recently to change the tar blocking? >> >> https://gerrit.ovirt.org/94357 >> >> I am leaving soon, perhaps someone can try the manual job with the >> result of the check-patch job for above patch, to see if it fixes. >> Otherwise I'll do this tomorrow. >> >> > >> > > >> > > >> > > 2018-09-14 15:49:23,032+0200 DEBUG otopi.context context._executeMethod:143 method exception >> > > Traceback (most recent call last): >> > > File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >> > > method['method']() >> > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/ovirt-host-deploy/core/misc.py", line 87, in _confirm >> > > prompt=True, >> > > File "/tmp/ovirt-O6CfS4aUHI/otopi-plugins/otopi/dialog/machine.py", line 478, in confirm >> > > code=opcode, >> > > >> > > >> > >> >> > >> On Fri, Sep 14, 2018 at 10:44 AM, Dafna Ron <dron@redhat.com> wrote: >> > >>> >> > >>> if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock >> > >>> >> > >>> >> > >>> On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina < mperina@redhat.com> wrote: >> > >>>> >> > >>>> >> > >>>> >> > >>>> On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote: >> > >>>>> >> > >>>>> did you use mock to reproduce? >> > >>>> >> > >>>> >> > >>>> No, just run_suite under myself >> > >>>>> >> > >>>>> >> > >>>>> On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina < mperina@redhat.com> wrote: >> > >>>>>> >> > >>>>>> Hi, >> > >>>>>> >> > >>>>>> the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running). >> > >>>>>> >> > >>>>>> So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it. >> > >>>>>> >> > >>>>>> Thanks >> > >>>>>> >> > >>>>>> Martin >> > >>>>>> >> > >>>>>> >> > >>>>>> On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron < dron@redhat.com> wrote: >> > >>>>>>> >> > >>>>>>> Full logs can be found here: >> > >>>>>>> >> > >>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >> > >>>>>>> >> > >>>>>>> On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron < dron@redhat.com> wrote: >> > >>>>>>>> >> > >>>>>>>> Hi, >> > >>>>>>>> >> > >>>>>>>> The previous regression was resolved and we now have a new regression. >> > >>>>>>>> >> > >>>>>>>> I don't think that the reported change is related so can someone from ovirt-engine take a look? >> > >>>>>>>> >> > >>>>>>>> The failure is add host on the upgrade suite. >> > >>>>>>>> >> > >>>>>>>> Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine. >> > >>>>>>>> >> > >>>>>>>> error: >> > >>>>>>>> >> > >>>>>>>> 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' >> > >>>>>>>> 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >> > >>>>>>>> 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back >> > >>>>>>>> 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog >> > >>>>>>>> 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >> > >>>>>>>> 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 >> > >>>>>>>> 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >> > >>>>>>>> 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 >> > >>>>>>>> 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. >> > >>>>>>>> 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' >> > >>>>>>>> 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) >> > >>>>>>>> (END) >> > >>>>>>>> >> > >>>>>>>> Thanks, >> > >>>>>>>> Dafna >> > >>>>>>>> >> > >>>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> -- >> > >>>>>> Martin Perina >> > >>>>>> Associate Manager, Software Engineering >> > >>>>>> Red Hat Czech s.r.o. >> > >>>>> >> > >>>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> Martin Perina >> > >>>> Associate Manager, Software Engineering >> > >>>> Red Hat Czech s.r.o. >> > >>> >> > >>> >> > >> >> > > >> > > >> > > >> > > -- >> > > Martin Perina >> > > Associate Manager, Software Engineering >> > > Red Hat Czech s.r.o. >> > >> > >> > >> > -- >> > Didi >> >> >> >> -- >> Didi
-- Didi
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/CG2IYPXSSEFTL6...
--
GALIT ROSENTHAL
SOFTWARE ENGINEER
Red Hat
galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/QMRM2INTCRDPT7...
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/45YRF2S6Q4Q4RV...
On Fri, Sep 14, 2018 at 4:44 PM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
Out of curiosity how to use mock_runner with run_suite? There are not mentioned any steps on execution using mock_runner in docs [1] (only installation of mock_runner), following command doesn't work: ./jenkins/mock_configs/mock_runner.sh -C ../jenkins/mock_configs -p el7 -e run_suite.sh basic-suite-master And I haven't found any parameters in mock_runner how to pass additional command line options to executed script. [1] https://ovirt-system-tests.readthedocs.io/en/latest/general/running_tests/in...
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi rt-master_change-queue-tester/10307/artifact/upgrade-from-re lease-suite.el7.x86_64/test_logs/upgrade-from-release-suite- master/post-002_bootstrap.py/
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
Hi,
The previous regression was resolved and we now have a new regression.
I don't think that the reported change is related so can someone from ovirt-engine take a look?
The failure is add host on the upgrade suite.
Please note that we have not had an engine-ovirt build for over 10 days due to several consecutive regressions and I would ask you to stop merging until we can stabilize the project and have a new package of engine.
error:
2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) [5c91fcbd] transaction rolled back 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] Error during deploy dialog 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during installation of Host lago-upgrade-from-release-suite-master-host-0: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host lago-upgrade-from-release-suite-master-host-0 install, preferring first exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', status='InstallFailed', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, SetVdsStatusVDSCommand, return: , log id: 146cdc08 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 installation failed. CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00. 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. (19 msec) (END)
Thanks, Dafna
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
Hi Martin, This is what I did. Checked out jenkins, ovirt-system-tests and ran the mock from ovirt-engine dir. ../jenkins/mock_configs/mock_runner.sh -e ../ovirt-system-tests/automation/upgrade-from-release_suite_master.sh el7 On Fri, Sep 14, 2018 at 11:23 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 4:44 PM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
Out of curiosity how to use mock_runner with run_suite? There are not mentioned any steps on execution using mock_runner in docs [1] (only installation of mock_runner), following command doesn't work:
./jenkins/mock_configs/mock_runner.sh -C ../jenkins/mock_configs -p el7 -e run_suite.sh basic-suite-master
And I haven't found any parameters in mock_runner how to pass additional command line options to executed script.
[1] https://ovirt-system-tests.readthedocs.io/en/latest/ general/running_tests/index.html
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
Full logs can be found here:
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovi rt-master_change-queue-tester/10307/artifact/upgrade-from-re lease-suite.el7.x86_64/test_logs/upgrade-from-release-suite- master/post-002_bootstrap.py/
On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote:
> Hi, > > The previous regression was resolved and we now have a new > regression. > > I don't think that the reported change is related so can someone > from ovirt-engine take a look? > > The failure is add host on the upgrade suite. > > Please note that we have not had an engine-ovirt build for over 10 > days due to several consecutive regressions and I would ask you to stop > merging until we can stabilize the project and have a new package of > engine. > > error: > > 2018-09-14 05:51:07,670-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute > 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; > MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; > trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > > /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && > "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine > DIALOG/customization=bool:True' > 2018-09-14 05:51:08,550-04 INFO [org.ovirt.engine.core.dal.dbb > roker.auditloghandling.AuditLogDirector] (VdsDeploy) [5c91fcbd] > EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing Host > lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. > 2018-09-14 05:51:08,565-04 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] > (VdsDeploy) [5c91fcbd] transaction rolled back > 2018-09-14 05:51:08,574-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (VdsDeploy) [5c91fcbd] Error during deploy dialog > 2018-09-14 05:51:08,578-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during > host lago-upgrade-from-release-suite-master-host-0 install > 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.dal.dbb > roker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: > VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during > installation of Host lago-upgrade-from-release-suite-master-host-0: > CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested > exception is org.postgresql.util.PSQLException: ERROR: invalid byte > sequence for encoding "UTF8": 0x00. > 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during > host lago-upgrade-from-release-suite-master-host-0 install, > preferring first exception: CallableStatementCallback; SQL [{call > insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for > encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: > ERROR: invalid byte sequence for encoding "UTF8": 0x00 > 2018-09-14 05:51:08,586-04 ERROR [org.ovirt.engine.core.bll.hos > tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) > [5c91fcbd] Host installation failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', > 'lago-upgrade-from-release-suite-master-host-0': > CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested > exception is org.postgresql.util.PSQLException: ERROR: invalid byte > sequence for encoding "UTF8": 0x00 > 2018-09-14 05:51:08,615-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, > SetVdsStatusVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, > SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', > status='InstallFailed', nonOperationalReason='NONE', > stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 > 2018-09-14 05:51:08,626-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, > SetVdsStatusVDSCommand, return: , log id: 146cdc08 > 2018-09-14 05:51:08,639-04 ERROR [org.ovirt.engine.core.dal.dbb > roker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: > VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 > installation failed. CallableStatementCallback; SQL [{call > insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, > ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for > encoding "UTF8": 0x00; nested exception is org.postgresql.util.PSQLException: > ERROR: invalid byte sequence for encoding "UTF8": 0x00. > 2018-09-14 05:51:08,652-04 INFO [org.ovirt.engine.core.bll.hos > tdeploy.InstallVdsInternalCommand] (EE-ManagedThreadFactory-engine-Thread-1) > [5c91fcbd] Lock freed to object 'EngineLock:{exclusiveLocks='[ > e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', sharedLocks=''}' > 2018-09-14 05:51:37,996-04 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] > (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache > updated. (19 msec) > (END) > > Thanks, > Dafna > >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
בתאריך יום ו׳, 14 בספט׳ 2018, 19:03, מאת Ravi Shankar Nori < rnori@redhat.com>:
Hi Martin,
This is what I did. Checked out jenkins, ovirt-system-tests and ran the mock from ovirt-engine dir.
../jenkins/mock_configs/mock_runner.sh -e ../ovirt-system-tests/automation/upgrade-from-release_suite_master.sh el7
You should actually run it from the ovirt-system-tests directory. I'm very much surprised running from the engine directory did anything useful, as it makes no sense...
On Fri, Sep 14, 2018 at 11:23 AM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 4:44 PM, Dafna Ron <dron@redhat.com> wrote:
if you run it with mock you would remove any environmental conditions that can effect the outcome so I recommend using mock
Out of curiosity how to use mock_runner with run_suite? There are not mentioned any steps on execution using mock_runner in docs [1] (only installation of mock_runner), following command doesn't work:
./jenkins/mock_configs/mock_runner.sh -C ../jenkins/mock_configs -p el7 -e run_suite.sh basic-suite-master
And I haven't found any parameters in mock_runner how to pass additional command line options to executed script.
[1] https://ovirt-system-tests.readthedocs.io/en/latest/general/running_tests/in...
On Fri, Sep 14, 2018 at 3:32 PM, Martin Perina <mperina@redhat.com> wrote:
On Fri, Sep 14, 2018 at 3:49 PM, Dafna Ron <dron@redhat.com> wrote:
did you use mock to reproduce?
No, just run_suite under myself
On Fri, Sep 14, 2018 at 2:39 PM, Martin Perina <mperina@redhat.com> wrote:
Hi,
the problem is that we haven't fetched the temporary host-deploy log from /tmp directory, so we don't know which string that host-deploy process sent to engine is causing that issue. I tried to reproduce on my local machine, but I was unable to reproduce it, 002_bootstrap phase finished successfully (other phases are still running).
So if anyone is able to reproduce, please try to fetch host-deploy log from /tmp directory after the error is raised and share it.
Thanks
Martin
On Fri, Sep 14, 2018 at 1:52 PM, Dafna Ron <dron@redhat.com> wrote:
> Full logs can be found here: > > > https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... > > On Fri, Sep 14, 2018 at 12:48 PM, Dafna Ron <dron@redhat.com> wrote: > >> Hi, >> >> The previous regression was resolved and we now have a new >> regression. >> >> I don't think that the reported change is related so can someone >> from ovirt-engine take a look? >> >> The failure is add host on the upgrade suite. >> >> Please note that we have not had an engine-ovirt build for over 10 >> days due to several consecutive regressions and I would ask you to stop >> merging until we can stabilize the project and have a new package of >> engine. >> >> error: >> >> 2018-09-14 05:51:07,670-04 INFO >> [org.ovirt.engine.core.uutils.ssh.SSHDialog] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] SSH execute >> 'root@lago-upgrade-from-release-suite-master-host-0' 'umask 0077; >> MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap >> "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > >> /dev/null 2>&1" 0; tar -b1 --warning=no-timestamp -C "${MYTMP}" -x && >> "${MYTMP}"/ovirt-host-deploy DIALOG/dialect=str:machine >> DIALOG/customization=bool:True' >> 2018-09-14 05:51:08,550-04 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (VdsDeploy) [5c91fcbd] EVENT_ID: VDS_INSTALL_IN_PROGRESS(509), Installing >> Host lago-upgrade-from-release-suite-master-host-0. Stage: Initializing. >> 2018-09-14 05:51:08,565-04 INFO >> [org.ovirt.engine.core.utils.transaction.TransactionSupport] (VdsDeploy) >> [5c91fcbd] transaction rolled back >> 2018-09-14 05:51:08,574-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [5c91fcbd] >> Error during deploy dialog >> 2018-09-14 05:51:08,578-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >> lago-upgrade-from-release-suite-master-host-0 install >> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >> VDS_INSTALL_IN_PROGRESS_ERROR(511), An error has occurred during >> installation of Host lago-upgrade-from-release-suite-master-host-0: >> CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; nested >> exception is org.postgresql.util.PSQLException: ERROR: invalid byte >> sequence for encoding "UTF8": 0x00. >> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Error during host >> lago-upgrade-from-release-suite-master-host-0 install, preferring first >> exception: CallableStatementCallback; SQL [{call insertauditlog(?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for encoding "UTF8": 0x00; >> nested exception is org.postgresql.util.PSQLException: ERROR: invalid byte >> sequence for encoding "UTF8": 0x00 >> 2018-09-14 05:51:08,586-04 ERROR >> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Host installation >> failed for host 'e475e93a-63b3-4573-b242-162c2ed864f0', >> 'lago-upgrade-from-release-suite-master-host-0': CallableStatementCallback; >> SQL [{call insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte >> sequence for encoding "UTF8": 0x00; nested exception is >> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >> encoding "UTF8": 0x00 >> 2018-09-14 05:51:08,615-04 INFO >> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] START, >> SetVdsStatusVDSCommand(HostName = >> lago-upgrade-from-release-suite-master-host-0, >> SetVdsStatusVDSCommandParameters:{hostId='e475e93a-63b3-4573-b242-162c2ed864f0', >> status='InstallFailed', nonOperationalReason='NONE', >> stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 146cdc08 >> 2018-09-14 05:51:08,626-04 INFO >> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] FINISH, >> SetVdsStatusVDSCommand, return: , log id: 146cdc08 >> 2018-09-14 05:51:08,639-04 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] EVENT_ID: >> VDS_INSTALL_FAILED(505), Host lago-upgrade-from-release-suite-master-host-0 >> installation failed. CallableStatementCallback; SQL [{call >> insertauditlog(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, >> ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}ERROR: invalid byte sequence for >> encoding "UTF8": 0x00; nested exception is >> org.postgresql.util.PSQLException: ERROR: invalid byte sequence for >> encoding "UTF8": 0x00. >> 2018-09-14 05:51:08,652-04 INFO >> [org.ovirt.engine.core.bll.hostdeploy.InstallVdsInternalCommand] >> (EE-ManagedThreadFactory-engine-Thread-1) [5c91fcbd] Lock freed to object >> 'EngineLock:{exclusiveLocks='[e475e93a-63b3-4573-b242-162c2ed864f0=VDS]', >> sharedLocks=''}' >> 2018-09-14 05:51:37,996-04 INFO >> [org.ovirt.engine.core.bll.quota.QuotaManager] >> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] Quota Cache updated. >> (19 msec) >> (END) >> >> Thanks, >> Dafna >> >> >
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/4344BCIBISAQIN...
participants (9)
-
Andrej Krejcir -
Barak Korren -
Dafna Ron -
Galit Rosenthal -
Martin Perina -
Ravi Shankar Nori -
Sandro Bonazzola -
Yedidyah Bar David -
Yuval Turgeman