Hi, The issue is probably not caused by the patch. This log line means that the VM does not exist in the DB: 2019-05-07 06:02:04,215-04 WARN [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation of action 'MigrateMultipleVms' failed for user admin@internal-authz. Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND I will investigate more, why the VM is missing. On Tue, 7 May 2019 at 14:07, Dafna Ron <dron(a)redhat.com&gt; wrote: > Hi, > > We are failing test upgrade_hosts on upgrade-from-release-suite-master. > From the logs I can see that we are calling migrate vm when we have only > one host and the vm seem to have been shut down before the maintenance call > is issued. > > Can you please look into this? > > suspected patch reported as root cause by CQ is: > > https://gerrit.ovirt.org/#/c/98920/ - core: Add MigrateMultipleVms > command and use it for host maintenance > > > logs are found here: > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14021/artif... > > > I can see the issue is vm migration when putting host in maintenance: > > > 2019-05-07 06:02:04,170-04 INFO > [org.ovirt.engine.core.bll.MaintenanceVdsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) > [05592db2-f859-487b-b779-4b32eec5bab > 3] Running command: MaintenanceVdsCommand internal: true. Entities > affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDS > 2019-05-07 06:02:04,215-04 WARN > [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation > of action > 'MigrateMultipleVms' failed for user admin@internal-authz. Reasons: > ACTION_TYPE_FAILED_VMS_NOT_FOUND > 2019-05-07 06:02:04,221-04 ERROR > [org.ovirt.engine.core.bll.MaintenanceVdsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Failed to > migrate one or > more VMs. > 2019-05-07 06:02:04,227-04 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] EVEN > T_ID: VDS_MAINTENANCE_FAILED(17), Failed to switch Host > lago-upgrade-from-release-suite-master-host-0 to Maintenance mode. > 2019-05-07 06:02:04,239-04 INFO > [org.ovirt.engine.core.bll.ActivateVdsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock > Acquired to object 'Eng > ineLock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', > sharedLocks=''}' > 2019-05-07 06:02:04,242-04 INFO > [org.ovirt.engine.core.bll.ActivateVdsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Running > command: ActivateVds > Command internal: true. Entities affected : ID: > 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDSAction group MANIPULATE_HOST > with role type ADMIN > 2019-05-07 06:02:04,243-04 INFO > [org.ovirt.engine.core.bll.ActivateVdsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Before > acquiring lock in ord > er to prevent monitoring for host > 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' > 2019-05-07 06:02:04,243-04 INFO > [org.ovirt.engine.core.bll.ActivateVdsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock > acquired, from now a mo > nitoring of host will be skipped for host > 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' > 2019-05-07 06:02:04,252-04 INFO > [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] START, > SetVdsStatu > sVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, > SetVdsStatusVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', > status='Unassigned', n > onOperationalReason='NONE', stopSpmFailureLogged='false', > maintenanceReason='null'}), log id: 2c8aa211 > 2019-05-07 06:02:04,256-04 INFO > [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] FINISH, > SetVdsStat > usVDSCommand, return: , log id: 2c8aa211 > 2019-05-07 06:02:04,261-04 INFO > [org.ovirt.engine.core.bll.ActivateVdsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Activate > host finished. Lock > released. Monitoring can run now for host > 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' > 2019-05-07 06:02:04,265-04 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] EVEN > T_ID: VDS_ACTIVATE(16), Activation of host > lago-upgrade-from-release-suite-master-host-0 initiated by > admin@internal-authz. > 2019-05-07 06:02:04,266-04 INFO > [org.ovirt.engine.core.bll.ActivateVdsCommand] > (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock freed > to object 'Engine > Lock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', > sharedLocks=''}' > 2019-05-07 06:02:04,484-04 ERROR > [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCallback] > (EE-ManagedThreadFactory-engineScheduled-Thread-96) > [05592db2-f859-487b-b779-4b32 > eec5bab3] Host 'lago-upgrade-from-release-suite-master-host-0' failed to > move to maintenance mode. Upgrade process is terminated. > > I can see there was only one vm running: > > > drwxrwxr-x. 2 dron dron 1024 May 7 11:49 qemu > [dron@dron post-004_basic_sanity.py]$ ls -l > lago-upgrade-from-release-suite-master-host-0/_var_log/libvirt/qemu/ > total 6 > -rw-rw-r--. 1 dron dron 4466 May 7 10:12 vm-with-iface.log > > and I can see that there was an attempt to terminate it with an error > that it does not exist: > > > stroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', > vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0', > gracefully='false', reason='', ig > noreNoVm='false'}), log id: 24278e9b > 2019-05-07 06:01:41,082-04 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default > task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] START, DestroyVDSComma > nd(HostName = lago-upgrade-from-release-suite-master-host-0, > DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', > vmId='dfbd75e2-a9cb-4fca-8788-a169 > 54db4abf', secondsToWait='0', gracefully='false', reason='', > ignoreNoVm='false'}), log id: 78bba2f8 > 2019-05-07 06:01:42,090-04 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default > task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVDSComm > and, return: , log id: 78bba2f8 > 2019-05-07 06:01:42,090-04 INFO > [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (default task-1) > [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVmVDSCommand, r > eturn: , log id: 24278e9b > 2019-05-07 06:01:42,094-04 INFO > [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] > (ForkJoinPool-1-worker-4) [] VM 'dfbd75e2-a9cb-4fca-8788-a16954db4abf' was > reported > as Down on VDS > '38e1379b-c3b6-4a2e-91df-d1f346e414a9'(lago-upgrade-from-release-suite-master-host-0) > 2019-05-07 06:01:42,096-04 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] > (ForkJoinPool-1-worker-4) [] START, DestroyVDSCommand(HostName = > lago-upgrade- > from-release-suite-master-host-0, > DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', > vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0 > ', gracefully='false', reason='', ignoreNoVm='true'}), log id: 1dbd31eb > 2019-05-07 06:01:42,114-04 INFO > [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] > (ForkJoinPool-1-worker-4) [] Failed to destroy VM > 'dfbd75e2-a9cb-4fca-8788-a16 > 954db4abf' because VM does not exist, ignoring > > > >

thanks for the quick reply and investigation. Please update me if I can help any further and if you find the cause and have a patch let me know. Note that ovirt-engine project is broken and if we cannot find the cause relatively fast we should consider reverting the patch to allow a new package to be built in CQ with other changes that were submitted. Thanks, Dafna On Tue, May 7, 2019 at 4:42 PM Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: > After running a few OSTs manually, it seems that the patch is the cause. > Investigating... > > On Tue, 7 May 2019 at 14:58, Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: > >> Hi, >> >> The issue is probably not caused by the patch. >> >> This log line means that the VM does not exist in the DB: >> >> 2019-05-07 06:02:04,215-04 WARN >> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >> of action 'MigrateMultipleVms' failed for user admin@internal-authz. >> Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND >> >> I will investigate more, why the VM is missing. >> >> On Tue, 7 May 2019 at 14:07, Dafna Ron <dron(a)redhat.com&gt; wrote: >> >>> Hi, >>> >>> We are failing test upgrade_hosts on upgrade-from-release-suite-master. >>> From the logs I can see that we are calling migrate vm when we have >>> only one host and the vm seem to have been shut down before the maintenance >>> call is issued. >>> >>> Can you please look into this? >>> >>> suspected patch reported as root cause by CQ is: >>> >>> https://gerrit.ovirt.org/#/c/98920/ - core: Add MigrateMultipleVms >>> command and use it for host maintenance >>> >>> >>> logs are found here: >>> >>> >>> >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14021/artif... >>> >>> >>> I can see the issue is vm migration when putting host in maintenance: >>> >>> >>> 2019-05-07 06:02:04,170-04 INFO >>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) >>> [05592db2-f859-487b-b779-4b32eec5bab >>> 3] Running command: MaintenanceVdsCommand internal: true. Entities >>> affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDS >>> 2019-05-07 06:02:04,215-04 WARN >>> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >>> of action >>> 'MigrateMultipleVms' failed for user admin@internal-authz. Reasons: >>> ACTION_TYPE_FAILED_VMS_NOT_FOUND >>> 2019-05-07 06:02:04,221-04 ERROR >>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Failed to >>> migrate one or >>> more VMs. >>> 2019-05-07 06:02:04,227-04 ERROR >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] EVEN >>> T_ID: VDS_MAINTENANCE_FAILED(17), Failed to switch Host >>> lago-upgrade-from-release-suite-master-host-0 to Maintenance mode. >>> 2019-05-07 06:02:04,239-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>> Acquired to object 'Eng >>> ineLock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>> sharedLocks=''}' >>> 2019-05-07 06:02:04,242-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Running >>> command: ActivateVds >>> Command internal: true. Entities affected : ID: >>> 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDSAction group MANIPULATE_HOST >>> with role type ADMIN >>> 2019-05-07 06:02:04,243-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Before >>> acquiring lock in ord >>> er to prevent monitoring for host >>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>> 2019-05-07 06:02:04,243-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>> acquired, from now a mo >>> nitoring of host will be skipped for host >>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>> 2019-05-07 06:02:04,252-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] START, >>> SetVdsStatu >>> sVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, >>> SetVdsStatusVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>> status='Unassigned', n >>> onOperationalReason='NONE', stopSpmFailureLogged='false', >>> maintenanceReason='null'}), log id: 2c8aa211 >>> 2019-05-07 06:02:04,256-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] FINISH, >>> SetVdsStat >>> usVDSCommand, return: , log id: 2c8aa211 >>> 2019-05-07 06:02:04,261-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Activate >>> host finished. Lock >>> released. Monitoring can run now for host >>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>> 2019-05-07 06:02:04,265-04 INFO >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] EVEN >>> T_ID: VDS_ACTIVATE(16), Activation of host >>> lago-upgrade-from-release-suite-master-host-0 initiated by >>> admin@internal-authz. >>> 2019-05-07 06:02:04,266-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock freed >>> to object 'Engine >>> Lock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>> sharedLocks=''}' >>> 2019-05-07 06:02:04,484-04 ERROR >>> [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCallback] >>> (EE-ManagedThreadFactory-engineScheduled-Thread-96) >>> [05592db2-f859-487b-b779-4b32 >>> eec5bab3] Host 'lago-upgrade-from-release-suite-master-host-0' failed >>> to move to maintenance mode. Upgrade process is terminated. >>> >>> I can see there was only one vm running: >>> >>> >>> drwxrwxr-x. 2 dron dron 1024 May 7 11:49 qemu >>> [dron@dron post-004_basic_sanity.py]$ ls -l >>> lago-upgrade-from-release-suite-master-host-0/_var_log/libvirt/qemu/ >>> total 6 >>> -rw-rw-r--. 1 dron dron 4466 May 7 10:12 vm-with-iface.log >>> >>> and I can see that there was an attempt to terminate it with an error >>> that it does not exist: >>> >>> >>> stroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0', >>> gracefully='false', reason='', ig >>> noreNoVm='false'}), log id: 24278e9b >>> 2019-05-07 06:01:41,082-04 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] START, DestroyVDSComma >>> nd(HostName = lago-upgrade-from-release-suite-master-host-0, >>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>> vmId='dfbd75e2-a9cb-4fca-8788-a169 >>> 54db4abf', secondsToWait='0', gracefully='false', reason='', >>> ignoreNoVm='false'}), log id: 78bba2f8 >>> 2019-05-07 06:01:42,090-04 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVDSComm >>> and, return: , log id: 78bba2f8 >>> 2019-05-07 06:01:42,090-04 INFO >>> [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (default task-1) >>> [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVmVDSCommand, r >>> eturn: , log id: 24278e9b >>> 2019-05-07 06:01:42,094-04 INFO >>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >>> (ForkJoinPool-1-worker-4) [] VM 'dfbd75e2-a9cb-4fca-8788-a16954db4abf' was >>> reported >>> as Down on VDS >>> '38e1379b-c3b6-4a2e-91df-d1f346e414a9'(lago-upgrade-from-release-suite-master-host-0) >>> 2019-05-07 06:01:42,096-04 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>> (ForkJoinPool-1-worker-4) [] START, DestroyVDSCommand(HostName = >>> lago-upgrade- >>> from-release-suite-master-host-0, >>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0 >>> ', gracefully='false', reason='', ignoreNoVm='true'}), log id: 1dbd31eb >>> 2019-05-07 06:01:42,114-04 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>> (ForkJoinPool-1-worker-4) [] Failed to destroy VM >>> 'dfbd75e2-a9cb-4fca-8788-a16 >>> 954db4abf' because VM does not exist, ignoring >>> >>> >>> >>>

Hi, I have not heard back on this issue and ovirt-engine has been broken for the past 3 days. As this does not seem a simple debug and fix I suggest reverting the patch and investigating later. thanks, Dafna On Wed, May 8, 2019 at 9:42 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > Any news? > > Thanks, > Dafna > > > On Tue, May 7, 2019 at 4:57 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> thanks for the quick reply and investigation. >> Please update me if I can help any further and if you find the cause and >> have a patch let me know. >> Note that ovirt-engine project is broken and if we cannot find the cause >> relatively fast we should consider reverting the patch to allow a new >> package to be built in CQ with other changes that were submitted. >> >> Thanks, >> Dafna >> >> >> On Tue, May 7, 2019 at 4:42 PM Andrej Krejcir <akrejcir(a)redhat.com&gt; >> wrote: >> >>> After running a few OSTs manually, it seems that the patch is the >>> cause. Investigating... >>> >>> On Tue, 7 May 2019 at 14:58, Andrej Krejcir <akrejcir(a)redhat.com&gt; >>> wrote: >>> >>>> Hi, >>>> >>>> The issue is probably not caused by the patch. >>>> >>>> This log line means that the VM does not exist in the DB: >>>> >>>> 2019-05-07 06:02:04,215-04 WARN >>>> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >>>> of action 'MigrateMultipleVms' failed for user admin@internal-authz. >>>> Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND >>>> >>>> I will investigate more, why the VM is missing. >>>> >>>> On Tue, 7 May 2019 at 14:07, Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>>> Hi, >>>>> >>>>> We are failing test upgrade_hosts on >>>>> upgrade-from-release-suite-master. >>>>> From the logs I can see that we are calling migrate vm when we have >>>>> only one host and the vm seem to have been shut down before the maintenance >>>>> call is issued. >>>>> >>>>> Can you please look into this? >>>>> >>>>> suspected patch reported as root cause by CQ is: >>>>> >>>>> https://gerrit.ovirt.org/#/c/98920/ - core: Add MigrateMultipleVms >>>>> command and use it for host maintenance >>>>> >>>>> >>>>> logs are found here: >>>>> >>>>> >>>>> >>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14021/artif... >>>>> >>>>> >>>>> I can see the issue is vm migration when putting host in maintenance: >>>>> >>>>> >>>>> 2019-05-07 06:02:04,170-04 INFO >>>>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) >>>>> [05592db2-f859-487b-b779-4b32eec5bab >>>>> 3] Running command: MaintenanceVdsCommand internal: true. Entities >>>>> affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDS >>>>> 2019-05-07 06:02:04,215-04 WARN >>>>> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >>>>> of action >>>>> 'MigrateMultipleVms' failed for user admin@internal-authz. Reasons: >>>>> ACTION_TYPE_FAILED_VMS_NOT_FOUND >>>>> 2019-05-07 06:02:04,221-04 ERROR >>>>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Failed to >>>>> migrate one or >>>>> more VMs. >>>>> 2019-05-07 06:02:04,227-04 ERROR >>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] EVEN >>>>> T_ID: VDS_MAINTENANCE_FAILED(17), Failed to switch Host >>>>> lago-upgrade-from-release-suite-master-host-0 to Maintenance mode. >>>>> 2019-05-07 06:02:04,239-04 INFO >>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>>>> Acquired to object 'Eng >>>>> ineLock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>>>> sharedLocks=''}' >>>>> 2019-05-07 06:02:04,242-04 INFO >>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Running >>>>> command: ActivateVds >>>>> Command internal: true. Entities affected : ID: >>>>> 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDSAction group MANIPULATE_HOST >>>>> with role type ADMIN >>>>> 2019-05-07 06:02:04,243-04 INFO >>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Before >>>>> acquiring lock in ord >>>>> er to prevent monitoring for host >>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>> 2019-05-07 06:02:04,243-04 INFO >>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>>>> acquired, from now a mo >>>>> nitoring of host will be skipped for host >>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>> 2019-05-07 06:02:04,252-04 INFO >>>>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] START, >>>>> SetVdsStatu >>>>> sVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, >>>>> SetVdsStatusVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>> status='Unassigned', n >>>>> onOperationalReason='NONE', stopSpmFailureLogged='false', >>>>> maintenanceReason='null'}), log id: 2c8aa211 >>>>> 2019-05-07 06:02:04,256-04 INFO >>>>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] FINISH, >>>>> SetVdsStat >>>>> usVDSCommand, return: , log id: 2c8aa211 >>>>> 2019-05-07 06:02:04,261-04 INFO >>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Activate >>>>> host finished. Lock >>>>> released. Monitoring can run now for host >>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>> 2019-05-07 06:02:04,265-04 INFO >>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] EVEN >>>>> T_ID: VDS_ACTIVATE(16), Activation of host >>>>> lago-upgrade-from-release-suite-master-host-0 initiated by >>>>> admin@internal-authz. >>>>> 2019-05-07 06:02:04,266-04 INFO >>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock freed >>>>> to object 'Engine >>>>> Lock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>>>> sharedLocks=''}' >>>>> 2019-05-07 06:02:04,484-04 ERROR >>>>> [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCallback] >>>>> (EE-ManagedThreadFactory-engineScheduled-Thread-96) >>>>> [05592db2-f859-487b-b779-4b32 >>>>> eec5bab3] Host 'lago-upgrade-from-release-suite-master-host-0' failed >>>>> to move to maintenance mode. Upgrade process is terminated. >>>>> >>>>> I can see there was only one vm running: >>>>> >>>>> >>>>> drwxrwxr-x. 2 dron dron 1024 May 7 11:49 qemu >>>>> [dron@dron post-004_basic_sanity.py]$ ls -l >>>>> lago-upgrade-from-release-suite-master-host-0/_var_log/libvirt/qemu/ >>>>> total 6 >>>>> -rw-rw-r--. 1 dron dron 4466 May 7 10:12 vm-with-iface.log >>>>> >>>>> and I can see that there was an attempt to terminate it with an error >>>>> that it does not exist: >>>>> >>>>> >>>>> stroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0', >>>>> gracefully='false', reason='', ig >>>>> noreNoVm='false'}), log id: 24278e9b >>>>> 2019-05-07 06:01:41,082-04 INFO >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>>>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] START, DestroyVDSComma >>>>> nd(HostName = lago-upgrade-from-release-suite-master-host-0, >>>>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>> vmId='dfbd75e2-a9cb-4fca-8788-a169 >>>>> 54db4abf', secondsToWait='0', gracefully='false', reason='', >>>>> ignoreNoVm='false'}), log id: 78bba2f8 >>>>> 2019-05-07 06:01:42,090-04 INFO >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>>>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVDSComm >>>>> and, return: , log id: 78bba2f8 >>>>> 2019-05-07 06:01:42,090-04 INFO >>>>> [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (default task-1) >>>>> [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVmVDSCommand, r >>>>> eturn: , log id: 24278e9b >>>>> 2019-05-07 06:01:42,094-04 INFO >>>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >>>>> (ForkJoinPool-1-worker-4) [] VM 'dfbd75e2-a9cb-4fca-8788-a16954db4abf' was >>>>> reported >>>>> as Down on VDS >>>>> '38e1379b-c3b6-4a2e-91df-d1f346e414a9'(lago-upgrade-from-release-suite-master-host-0) >>>>> 2019-05-07 06:01:42,096-04 INFO >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>>>> (ForkJoinPool-1-worker-4) [] START, DestroyVDSCommand(HostName = >>>>> lago-upgrade- >>>>> from-release-suite-master-host-0, >>>>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0 >>>>> ', gracefully='false', reason='', ignoreNoVm='true'}), log id: >>>>> 1dbd31eb >>>>> 2019-05-07 06:01:42,114-04 INFO >>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>>>> (ForkJoinPool-1-worker-4) [] Failed to destroy VM >>>>> 'dfbd75e2-a9cb-4fca-8788-a16 >>>>> 954db4abf' because VM does not exist, ignoring >>>>> >>>>> >>>>> >>>>>

Thanks Andrej. I will follow the patch and update. Dafna On Thu, May 9, 2019 at 11:23 AM Andrej Krejcir <akrejcir(a)redhat.com&gt; wrote: > Hi, > > Ok, I have posted the reverting patch: > https://gerrit.ovirt.org/#/c/99845/ > > I'm still investigating what is the problem. Sorry for the delay, we had > a public holiday yesturday. > > > Andrej > > On Thu, 9 May 2019 at 11:20, Dafna Ron <dron(a)redhat.com&gt; wrote: > >> Hi, >> >> I have not heard back on this issue and ovirt-engine has been broken for >> the past 3 days. >> >> As this does not seem a simple debug and fix I suggest reverting the >> patch and investigating later. >> >> thanks, >> Dafna >> >> >> >> On Wed, May 8, 2019 at 9:42 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >> >>> Any news? >>> >>> Thanks, >>> Dafna >>> >>> >>> On Tue, May 7, 2019 at 4:57 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>> >>>> thanks for the quick reply and investigation. >>>> Please update me if I can help any further and if you find the cause >>>> and have a patch let me know. >>>> Note that ovirt-engine project is broken and if we cannot find the >>>> cause relatively fast we should consider reverting the patch to allow a new >>>> package to be built in CQ with other changes that were submitted. >>>> >>>> Thanks, >>>> Dafna >>>> >>>> >>>> On Tue, May 7, 2019 at 4:42 PM Andrej Krejcir <akrejcir(a)redhat.com&gt; >>>> wrote: >>>> >>>>> After running a few OSTs manually, it seems that the patch is the >>>>> cause. Investigating... >>>>> >>>>> On Tue, 7 May 2019 at 14:58, Andrej Krejcir <akrejcir(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> The issue is probably not caused by the patch. >>>>>> >>>>>> This log line means that the VM does not exist in the DB: >>>>>> >>>>>> 2019-05-07 06:02:04,215-04 WARN >>>>>> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >>>>>> of action 'MigrateMultipleVms' failed for user admin@internal-authz. >>>>>> Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND >>>>>> >>>>>> I will investigate more, why the VM is missing. >>>>>> >>>>>> On Tue, 7 May 2019 at 14:07, Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> We are failing test upgrade_hosts on >>>>>>> upgrade-from-release-suite-master. >>>>>>> From the logs I can see that we are calling migrate vm when we have >>>>>>> only one host and the vm seem to have been shut down before the maintenance >>>>>>> call is issued. >>>>>>> >>>>>>> Can you please look into this? >>>>>>> >>>>>>> suspected patch reported as root cause by CQ is: >>>>>>> >>>>>>> https://gerrit.ovirt.org/#/c/98920/ - core: Add MigrateMultipleVms >>>>>>> command and use it for host maintenance >>>>>>> >>>>>>> >>>>>>> logs are found here: >>>>>>> >>>>>>> >>>>>>> >>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14021/artif... >>>>>>> >>>>>>> >>>>>>> I can see the issue is vm migration when putting host in >>>>>>> maintenance: >>>>>>> >>>>>>> >>>>>>> 2019-05-07 06:02:04,170-04 INFO >>>>>>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) >>>>>>> [05592db2-f859-487b-b779-4b32eec5bab >>>>>>> 3] Running command: MaintenanceVdsCommand internal: true. Entities >>>>>>> affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDS >>>>>>> 2019-05-07 06:02:04,215-04 WARN >>>>>>> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >>>>>>> of action >>>>>>> 'MigrateMultipleVms' failed for user admin@internal-authz. >>>>>>> Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND >>>>>>> 2019-05-07 06:02:04,221-04 ERROR >>>>>>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Failed to >>>>>>> migrate one or >>>>>>> more VMs. >>>>>>> 2019-05-07 06:02:04,227-04 ERROR >>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] EVEN >>>>>>> T_ID: VDS_MAINTENANCE_FAILED(17), Failed to switch Host >>>>>>> lago-upgrade-from-release-suite-master-host-0 to Maintenance mode. >>>>>>> 2019-05-07 06:02:04,239-04 INFO >>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>>>>>> Acquired to object 'Eng >>>>>>> ineLock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>>>>>> sharedLocks=''}' >>>>>>> 2019-05-07 06:02:04,242-04 INFO >>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Running >>>>>>> command: ActivateVds >>>>>>> Command internal: true. Entities affected : ID: >>>>>>> 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDSAction group MANIPULATE_HOST >>>>>>> with role type ADMIN >>>>>>> 2019-05-07 06:02:04,243-04 INFO >>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Before >>>>>>> acquiring lock in ord >>>>>>> er to prevent monitoring for host >>>>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>>>> 2019-05-07 06:02:04,243-04 INFO >>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>>>>>> acquired, from now a mo >>>>>>> nitoring of host will be skipped for host >>>>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>>>> 2019-05-07 06:02:04,252-04 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] START, >>>>>>> SetVdsStatu >>>>>>> sVDSCommand(HostName = >>>>>>> lago-upgrade-from-release-suite-master-host-0, >>>>>>> SetVdsStatusVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>>>> status='Unassigned', n >>>>>>> onOperationalReason='NONE', stopSpmFailureLogged='false', >>>>>>> maintenanceReason='null'}), log id: 2c8aa211 >>>>>>> 2019-05-07 06:02:04,256-04 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] FINISH, >>>>>>> SetVdsStat >>>>>>> usVDSCommand, return: , log id: 2c8aa211 >>>>>>> 2019-05-07 06:02:04,261-04 INFO >>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Activate >>>>>>> host finished. Lock >>>>>>> released. Monitoring can run now for host >>>>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>>>> 2019-05-07 06:02:04,265-04 INFO >>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] EVEN >>>>>>> T_ID: VDS_ACTIVATE(16), Activation of host >>>>>>> lago-upgrade-from-release-suite-master-host-0 initiated by >>>>>>> admin@internal-authz. >>>>>>> 2019-05-07 06:02:04,266-04 INFO >>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock freed >>>>>>> to object 'Engine >>>>>>> Lock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>>>>>> sharedLocks=''}' >>>>>>> 2019-05-07 06:02:04,484-04 ERROR >>>>>>> [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCallback] >>>>>>> (EE-ManagedThreadFactory-engineScheduled-Thread-96) >>>>>>> [05592db2-f859-487b-b779-4b32 >>>>>>> eec5bab3] Host 'lago-upgrade-from-release-suite-master-host-0' >>>>>>> failed to move to maintenance mode. Upgrade process is terminated. >>>>>>> >>>>>>> I can see there was only one vm running: >>>>>>> >>>>>>> >>>>>>> drwxrwxr-x. 2 dron dron 1024 May 7 11:49 qemu >>>>>>> [dron@dron post-004_basic_sanity.py]$ ls -l >>>>>>> lago-upgrade-from-release-suite-master-host-0/_var_log/libvirt/qemu/ >>>>>>> total 6 >>>>>>> -rw-rw-r--. 1 dron dron 4466 May 7 10:12 vm-with-iface.log >>>>>>> >>>>>>> and I can see that there was an attempt to terminate it with an >>>>>>> error that it does not exist: >>>>>>> >>>>>>> >>>>>>> stroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>>>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0', >>>>>>> gracefully='false', reason='', ig >>>>>>> noreNoVm='false'}), log id: 24278e9b >>>>>>> 2019-05-07 06:01:41,082-04 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>>>>>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] START, DestroyVDSComma >>>>>>> nd(HostName = lago-upgrade-from-release-suite-master-host-0, >>>>>>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>>>> vmId='dfbd75e2-a9cb-4fca-8788-a169 >>>>>>> 54db4abf', secondsToWait='0', gracefully='false', reason='', >>>>>>> ignoreNoVm='false'}), log id: 78bba2f8 >>>>>>> 2019-05-07 06:01:42,090-04 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>>>>>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVDSComm >>>>>>> and, return: , log id: 78bba2f8 >>>>>>> 2019-05-07 06:01:42,090-04 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (default task-1) >>>>>>> [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVmVDSCommand, r >>>>>>> eturn: , log id: 24278e9b >>>>>>> 2019-05-07 06:01:42,094-04 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >>>>>>> (ForkJoinPool-1-worker-4) [] VM 'dfbd75e2-a9cb-4fca-8788-a16954db4abf' was >>>>>>> reported >>>>>>> as Down on VDS >>>>>>> '38e1379b-c3b6-4a2e-91df-d1f346e414a9'(lago-upgrade-from-release-suite-master-host-0) >>>>>>> 2019-05-07 06:01:42,096-04 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>>>>>> (ForkJoinPool-1-worker-4) [] START, DestroyVDSCommand(HostName = >>>>>>> lago-upgrade- >>>>>>> from-release-suite-master-host-0, >>>>>>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>>>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0 >>>>>>> ', gracefully='false', reason='', ignoreNoVm='true'}), log id: >>>>>>> 1dbd31eb >>>>>>> 2019-05-07 06:01:42,114-04 INFO >>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>>>>>> (ForkJoinPool-1-worker-4) [] Failed to destroy VM >>>>>>> 'dfbd75e2-a9cb-4fca-8788-a16 >>>>>>> 954db4abf' because VM does not exist, ignoring >>>>>>> >>>>>>> >>>>>>> >>>>>>>

Il giorno gio 9 mag 2019 alle ore 12:59 Dafna Ron <dron(a)redhat.com&gt; ha scritto: > As IL are on independence day, anyone else can merge? > https://gerrit.ovirt.org/#/c/99845/ > > I have merge rights but I need at least CI to pass. Waiting on jenkins. > > On Thu, May 9, 2019 at 11:30 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> Thanks Andrej. >> I will follow the patch and update. >> Dafna >> >> On Thu, May 9, 2019 at 11:23 AM Andrej Krejcir <akrejcir(a)redhat.com&gt; >> wrote: >> >>> Hi, >>> >>> Ok, I have posted the reverting patch: >>> https://gerrit.ovirt.org/#/c/99845/ >>> >>> I'm still investigating what is the problem. Sorry for the delay, we >>> had a public holiday yesturday. >>> >>> >>> Andrej >>> >>> On Thu, 9 May 2019 at 11:20, Dafna Ron <dron(a)redhat.com&gt; wrote: >>> >>>> Hi, >>>> >>>> I have not heard back on this issue and ovirt-engine has been broken >>>> for the past 3 days. >>>> >>>> As this does not seem a simple debug and fix I suggest reverting the >>>> patch and investigating later. >>>> >>>> thanks, >>>> Dafna >>>> >>>> >>>> >>>> On Wed, May 8, 2019 at 9:42 AM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>>> Any news? >>>>> >>>>> Thanks, >>>>> Dafna >>>>> >>>>> >>>>> On Tue, May 7, 2019 at 4:57 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>> >>>>>> thanks for the quick reply and investigation. >>>>>> Please update me if I can help any further and if you find the cause >>>>>> and have a patch let me know. >>>>>> Note that ovirt-engine project is broken and if we cannot find the >>>>>> cause relatively fast we should consider reverting the patch to allow a new >>>>>> package to be built in CQ with other changes that were submitted. >>>>>> >>>>>> Thanks, >>>>>> Dafna >>>>>> >>>>>> >>>>>> On Tue, May 7, 2019 at 4:42 PM Andrej Krejcir <akrejcir(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> After running a few OSTs manually, it seems that the patch is the >>>>>>> cause. Investigating... >>>>>>> >>>>>>> On Tue, 7 May 2019 at 14:58, Andrej Krejcir <akrejcir(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> The issue is probably not caused by the patch. >>>>>>>> >>>>>>>> This log line means that the VM does not exist in the DB: >>>>>>>> >>>>>>>> 2019-05-07 06:02:04,215-04 WARN >>>>>>>> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >>>>>>>> of action 'MigrateMultipleVms' failed for user admin@internal-authz. >>>>>>>> Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND >>>>>>>> >>>>>>>> I will investigate more, why the VM is missing. >>>>>>>> >>>>>>>> On Tue, 7 May 2019 at 14:07, Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> We are failing test upgrade_hosts on >>>>>>>>> upgrade-from-release-suite-master. >>>>>>>>> From the logs I can see that we are calling migrate vm when we >>>>>>>>> have only one host and the vm seem to have been shut down before the >>>>>>>>> maintenance call is issued. >>>>>>>>> >>>>>>>>> Can you please look into this? >>>>>>>>> >>>>>>>>> suspected patch reported as root cause by CQ is: >>>>>>>>> >>>>>>>>> https://gerrit.ovirt.org/#/c/98920/ - core: Add >>>>>>>>> MigrateMultipleVms command and use it for host maintenance >>>>>>>>> >>>>>>>>> >>>>>>>>> logs are found here: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14021/artif... >>>>>>>>> >>>>>>>>> >>>>>>>>> I can see the issue is vm migration when putting host in >>>>>>>>> maintenance: >>>>>>>>> >>>>>>>>> >>>>>>>>> 2019-05-07 06:02:04,170-04 INFO >>>>>>>>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) >>>>>>>>> [05592db2-f859-487b-b779-4b32eec5bab >>>>>>>>> 3] Running command: MaintenanceVdsCommand internal: true. >>>>>>>>> Entities affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDS >>>>>>>>> 2019-05-07 06:02:04,215-04 WARN >>>>>>>>> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >>>>>>>>> of action >>>>>>>>> 'MigrateMultipleVms' failed for user admin@internal-authz. >>>>>>>>> Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND >>>>>>>>> 2019-05-07 06:02:04,221-04 ERROR >>>>>>>>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Failed to >>>>>>>>> migrate one or >>>>>>>>> more VMs. >>>>>>>>> 2019-05-07 06:02:04,227-04 ERROR >>>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] EVEN >>>>>>>>> T_ID: VDS_MAINTENANCE_FAILED(17), Failed to switch Host >>>>>>>>> lago-upgrade-from-release-suite-master-host-0 to Maintenance mode. >>>>>>>>> 2019-05-07 06:02:04,239-04 INFO >>>>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>>>>>>>> Acquired to object 'Eng >>>>>>>>> ineLock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>>>>>>>> sharedLocks=''}' >>>>>>>>> 2019-05-07 06:02:04,242-04 INFO >>>>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Running >>>>>>>>> command: ActivateVds >>>>>>>>> Command internal: true. Entities affected : ID: >>>>>>>>> 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDSAction group MANIPULATE_HOST >>>>>>>>> with role type ADMIN >>>>>>>>> 2019-05-07 06:02:04,243-04 INFO >>>>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Before >>>>>>>>> acquiring lock in ord >>>>>>>>> er to prevent monitoring for host >>>>>>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>>>>>> 2019-05-07 06:02:04,243-04 INFO >>>>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>>>>>>>> acquired, from now a mo >>>>>>>>> nitoring of host will be skipped for host >>>>>>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>>>>>> 2019-05-07 06:02:04,252-04 INFO >>>>>>>>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] START, >>>>>>>>> SetVdsStatu >>>>>>>>> sVDSCommand(HostName = >>>>>>>>> lago-upgrade-from-release-suite-master-host-0, >>>>>>>>> SetVdsStatusVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>>>>>> status='Unassigned', n >>>>>>>>> onOperationalReason='NONE', stopSpmFailureLogged='false', >>>>>>>>> maintenanceReason='null'}), log id: 2c8aa211 >>>>>>>>> 2019-05-07 06:02:04,256-04 INFO >>>>>>>>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] FINISH, >>>>>>>>> SetVdsStat >>>>>>>>> usVDSCommand, return: , log id: 2c8aa211 >>>>>>>>> 2019-05-07 06:02:04,261-04 INFO >>>>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Activate >>>>>>>>> host finished. Lock >>>>>>>>> released. Monitoring can run now for host >>>>>>>>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>>>>>>>> 2019-05-07 06:02:04,265-04 INFO >>>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] EVEN >>>>>>>>> T_ID: VDS_ACTIVATE(16), Activation of host >>>>>>>>> lago-upgrade-from-release-suite-master-host-0 initiated by >>>>>>>>> admin@internal-authz. >>>>>>>>> 2019-05-07 06:02:04,266-04 INFO >>>>>>>>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>>>>>>>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock freed >>>>>>>>> to object 'Engine >>>>>>>>> Lock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>>>>>>>> sharedLocks=''}' >>>>>>>>> 2019-05-07 06:02:04,484-04 ERROR >>>>>>>>> [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCallback] >>>>>>>>> (EE-ManagedThreadFactory-engineScheduled-Thread-96) >>>>>>>>> [05592db2-f859-487b-b779-4b32 >>>>>>>>> eec5bab3] Host 'lago-upgrade-from-release-suite-master-host-0' >>>>>>>>> failed to move to maintenance mode. Upgrade process is terminated. >>>>>>>>> >>>>>>>>> I can see there was only one vm running: >>>>>>>>> >>>>>>>>> >>>>>>>>> drwxrwxr-x. 2 dron dron 1024 May 7 11:49 qemu >>>>>>>>> [dron@dron post-004_basic_sanity.py]$ ls -l >>>>>>>>> lago-upgrade-from-release-suite-master-host-0/_var_log/libvirt/qemu/ >>>>>>>>> total 6 >>>>>>>>> -rw-rw-r--. 1 dron dron 4466 May 7 10:12 vm-with-iface.log >>>>>>>>> >>>>>>>>> and I can see that there was an attempt to terminate it with an >>>>>>>>> error that it does not exist: >>>>>>>>> >>>>>>>>> >>>>>>>>> stroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>>>>>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0', >>>>>>>>> gracefully='false', reason='', ig >>>>>>>>> noreNoVm='false'}), log id: 24278e9b >>>>>>>>> 2019-05-07 06:01:41,082-04 INFO >>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>>>>>>>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] START, DestroyVDSComma >>>>>>>>> nd(HostName = lago-upgrade-from-release-suite-master-host-0, >>>>>>>>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>>>>>> vmId='dfbd75e2-a9cb-4fca-8788-a169 >>>>>>>>> 54db4abf', secondsToWait='0', gracefully='false', reason='', >>>>>>>>> ignoreNoVm='false'}), log id: 78bba2f8 >>>>>>>>> 2019-05-07 06:01:42,090-04 INFO >>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>>>>>>>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVDSComm >>>>>>>>> and, return: , log id: 78bba2f8 >>>>>>>>> 2019-05-07 06:01:42,090-04 INFO >>>>>>>>> [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (default task-1) >>>>>>>>> [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVmVDSCommand, r >>>>>>>>> eturn: , log id: 24278e9b >>>>>>>>> 2019-05-07 06:01:42,094-04 INFO >>>>>>>>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >>>>>>>>> (ForkJoinPool-1-worker-4) [] VM 'dfbd75e2-a9cb-4fca-8788-a16954db4abf' was >>>>>>>>> reported >>>>>>>>> as Down on VDS >>>>>>>>> '38e1379b-c3b6-4a2e-91df-d1f346e414a9'(lago-upgrade-from-release-suite-master-host-0) >>>>>>>>> 2019-05-07 06:01:42,096-04 INFO >>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>>>>>>>> (ForkJoinPool-1-worker-4) [] START, DestroyVDSCommand(HostName = >>>>>>>>> lago-upgrade- >>>>>>>>> from-release-suite-master-host-0, >>>>>>>>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>>>>>>>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0 >>>>>>>>> ', gracefully='false', reason='', ignoreNoVm='true'}), log id: >>>>>>>>> 1dbd31eb >>>>>>>>> 2019-05-07 06:01:42,114-04 INFO >>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>>>>>>>> (ForkJoinPool-1-worker-4) [] Failed to destroy VM >>>>>>>>> 'dfbd75e2-a9cb-4fca-8788-a16 >>>>>>>>> 954db4abf' because VM does not exist, ignoring >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/OQ22IENFIVB... > -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> <https://redhat.com/summit>