Il giorno gio 9 mag 2019 alle ore 12:59 Dafna Ron <dron@redhat.com> ha scritto:
As IL are on independence day, anyone else can merge? https://gerrit.ovirt.org/#/c/99845/
I have merge rights but I need at least CI to pass. Waiting on jenkins.
On Thu, May 9, 2019 at 11:30 AM Dafna Ron <dron@redhat.com> wrote:
Thanks Andrej. I will follow the patch and update. Dafna
On Thu, May 9, 2019 at 11:23 AM Andrej Krejcir <akrejcir@redhat.com> wrote:
Hi,
Ok, I have posted the reverting patch: https://gerrit.ovirt.org/#/c/99845/
I'm still investigating what is the problem. Sorry for the delay, we had a public holiday yesturday.
Andrej
On Thu, 9 May 2019 at 11:20, Dafna Ron <dron@redhat.com> wrote:
Hi,
I have not heard back on this issue and ovirt-engine has been broken for the past 3 days.
As this does not seem a simple debug and fix I suggest reverting the patch and investigating later.
thanks, Dafna
On Wed, May 8, 2019 at 9:42 AM Dafna Ron <dron@redhat.com> wrote:
Any news?
Thanks, Dafna
On Tue, May 7, 2019 at 4:57 PM Dafna Ron <dron@redhat.com> wrote:
thanks for the quick reply and investigation. Please update me if I can help any further and if you find the cause and have a patch let me know. Note that ovirt-engine project is broken and if we cannot find the cause relatively fast we should consider reverting the patch to allow a new package to be built in CQ with other changes that were submitted.
Thanks, Dafna
On Tue, May 7, 2019 at 4:42 PM Andrej Krejcir <akrejcir@redhat.com> wrote:
> After running a few OSTs manually, it seems that the patch is the > cause. Investigating... > > On Tue, 7 May 2019 at 14:58, Andrej Krejcir <akrejcir@redhat.com> > wrote: > >> Hi, >> >> The issue is probably not caused by the patch. >> >> This log line means that the VM does not exist in the DB: >> >> 2019-05-07 06:02:04,215-04 WARN >> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >> of action 'MigrateMultipleVms' failed for user admin@internal-authz. >> Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND >> >> I will investigate more, why the VM is missing. >> >> On Tue, 7 May 2019 at 14:07, Dafna Ron <dron@redhat.com> wrote: >> >>> Hi, >>> >>> We are failing test upgrade_hosts on >>> upgrade-from-release-suite-master. >>> From the logs I can see that we are calling migrate vm when we >>> have only one host and the vm seem to have been shut down before the >>> maintenance call is issued. >>> >>> Can you please look into this? >>> >>> suspected patch reported as root cause by CQ is: >>> >>> https://gerrit.ovirt.org/#/c/98920/ - core: Add >>> MigrateMultipleVms command and use it for host maintenance >>> >>> >>> logs are found here: >>> >>> >>> >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14021/artifact... >>> >>> >>> I can see the issue is vm migration when putting host in >>> maintenance: >>> >>> >>> 2019-05-07 06:02:04,170-04 INFO >>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) >>> [05592db2-f859-487b-b779-4b32eec5bab >>> 3] Running command: MaintenanceVdsCommand internal: true. Entities >>> affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDS >>> 2019-05-07 06:02:04,215-04 WARN >>> [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation >>> of action >>> 'MigrateMultipleVms' failed for user admin@internal-authz. >>> Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND >>> 2019-05-07 06:02:04,221-04 ERROR >>> [org.ovirt.engine.core.bll.MaintenanceVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Failed to >>> migrate one or >>> more VMs. >>> 2019-05-07 06:02:04,227-04 ERROR >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] EVEN >>> T_ID: VDS_MAINTENANCE_FAILED(17), Failed to switch Host >>> lago-upgrade-from-release-suite-master-host-0 to Maintenance mode. >>> 2019-05-07 06:02:04,239-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>> Acquired to object 'Eng >>> ineLock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>> sharedLocks=''}' >>> 2019-05-07 06:02:04,242-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Running >>> command: ActivateVds >>> Command internal: true. Entities affected : ID: >>> 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDSAction group MANIPULATE_HOST >>> with role type ADMIN >>> 2019-05-07 06:02:04,243-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Before >>> acquiring lock in ord >>> er to prevent monitoring for host >>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>> 2019-05-07 06:02:04,243-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock >>> acquired, from now a mo >>> nitoring of host will be skipped for host >>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>> 2019-05-07 06:02:04,252-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] START, >>> SetVdsStatu >>> sVDSCommand(HostName = >>> lago-upgrade-from-release-suite-master-host-0, >>> SetVdsStatusVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>> status='Unassigned', n >>> onOperationalReason='NONE', stopSpmFailureLogged='false', >>> maintenanceReason='null'}), log id: 2c8aa211 >>> 2019-05-07 06:02:04,256-04 INFO >>> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] FINISH, >>> SetVdsStat >>> usVDSCommand, return: , log id: 2c8aa211 >>> 2019-05-07 06:02:04,261-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Activate >>> host finished. Lock >>> released. Monitoring can run now for host >>> 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' >>> 2019-05-07 06:02:04,265-04 INFO >>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] EVEN >>> T_ID: VDS_ACTIVATE(16), Activation of host >>> lago-upgrade-from-release-suite-master-host-0 initiated by >>> admin@internal-authz. >>> 2019-05-07 06:02:04,266-04 INFO >>> [org.ovirt.engine.core.bll.ActivateVdsCommand] >>> (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock freed >>> to object 'Engine >>> Lock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', >>> sharedLocks=''}' >>> 2019-05-07 06:02:04,484-04 ERROR >>> [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCallback] >>> (EE-ManagedThreadFactory-engineScheduled-Thread-96) >>> [05592db2-f859-487b-b779-4b32 >>> eec5bab3] Host 'lago-upgrade-from-release-suite-master-host-0' >>> failed to move to maintenance mode. Upgrade process is terminated. >>> >>> I can see there was only one vm running: >>> >>> >>> drwxrwxr-x. 2 dron dron 1024 May 7 11:49 qemu >>> [dron@dron post-004_basic_sanity.py]$ ls -l >>> lago-upgrade-from-release-suite-master-host-0/_var_log/libvirt/qemu/ >>> total 6 >>> -rw-rw-r--. 1 dron dron 4466 May 7 10:12 vm-with-iface.log >>> >>> and I can see that there was an attempt to terminate it with an >>> error that it does not exist: >>> >>> >>> stroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0', >>> gracefully='false', reason='', ig >>> noreNoVm='false'}), log id: 24278e9b >>> 2019-05-07 06:01:41,082-04 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] START, DestroyVDSComma >>> nd(HostName = lago-upgrade-from-release-suite-master-host-0, >>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>> vmId='dfbd75e2-a9cb-4fca-8788-a169 >>> 54db4abf', secondsToWait='0', gracefully='false', reason='', >>> ignoreNoVm='false'}), log id: 78bba2f8 >>> 2019-05-07 06:01:42,090-04 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default >>> task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVDSComm >>> and, return: , log id: 78bba2f8 >>> 2019-05-07 06:01:42,090-04 INFO >>> [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (default task-1) >>> [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVmVDSCommand, r >>> eturn: , log id: 24278e9b >>> 2019-05-07 06:01:42,094-04 INFO >>> [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] >>> (ForkJoinPool-1-worker-4) [] VM 'dfbd75e2-a9cb-4fca-8788-a16954db4abf' was >>> reported >>> as Down on VDS >>> '38e1379b-c3b6-4a2e-91df-d1f346e414a9'(lago-upgrade-from-release-suite-master-host-0) >>> 2019-05-07 06:01:42,096-04 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>> (ForkJoinPool-1-worker-4) [] START, DestroyVDSCommand(HostName = >>> lago-upgrade- >>> from-release-suite-master-host-0, >>> DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', >>> vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0 >>> ', gracefully='false', reason='', ignoreNoVm='true'}), log id: >>> 1dbd31eb >>> 2019-05-07 06:01:42,114-04 INFO >>> [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] >>> (ForkJoinPool-1-worker-4) [] Failed to destroy VM >>> 'dfbd75e2-a9cb-4fca-8788-a16 >>> 954db4abf' because VM does not exist, ignoring >>> >>> >>> >>> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/OQ22IENFIVB2SH...
-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig> <https://redhat.com/summit>