- Devel - oVirt List Archives

Subject: [ OST Failure Report ] [ oVirt Master (ovirt-engine) ] [ 05-05-2019 ] [ upgrade_hosts ]
by Dafna Ron 10 May '19

10 May '19

Hi, We are failing test upgrade_hosts on upgrade-from-release-suite-master. From the logs I can see that we are calling migrate vm when we have only one host and the vm seem to have been shut down before the maintenance call is issued. Can you please look into this? suspected patch reported as root cause by CQ is: https://gerrit.ovirt.org/#/c/98920/ - core: Add MigrateMultipleVms command and use it for host maintenance logs are found here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/14021/artifac… I can see the issue is vm migration when putting host in maintenance: 2019-05-07 06:02:04,170-04 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [05592db2-f859-487b-b779-4b32eec5bab 3] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDS 2019-05-07 06:02:04,215-04 WARN [org.ovirt.engine.core.bll.MigrateMultipleVmsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Validation of action 'MigrateMultipleVms' failed for user admin@internal-authz. Reasons: ACTION_TYPE_FAILED_VMS_NOT_FOUND 2019-05-07 06:02:04,221-04 ERROR [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] Failed to migrate one or more VMs. 2019-05-07 06:02:04,227-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [33485140] EVEN T_ID: VDS_MAINTENANCE_FAILED(17), Failed to switch Host lago-upgrade-from-release-suite-master-host-0 to Maintenance mode. 2019-05-07 06:02:04,239-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock Acquired to object 'Eng ineLock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', sharedLocks=''}' 2019-05-07 06:02:04,242-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Running command: ActivateVds Command internal: true. Entities affected : ID: 38e1379b-c3b6-4a2e-91df-d1f346e414a9 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-05-07 06:02:04,243-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Before acquiring lock in ord er to prevent monitoring for host 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' 2019-05-07 06:02:04,243-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock acquired, from now a mo nitoring of host will be skipped for host 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' 2019-05-07 06:02:04,252-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] START, SetVdsStatu sVDSCommand(HostName = lago-upgrade-from-release-suite-master-host-0, SetVdsStatusVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', status='Unassigned', n onOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2c8aa211 2019-05-07 06:02:04,256-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] FINISH, SetVdsStat usVDSCommand, return: , log id: 2c8aa211 2019-05-07 06:02:04,261-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Activate host finished. Lock released. Monitoring can run now for host 'lago-upgrade-from-release-suite-master-host-0' from data-center 'test-dc' 2019-05-07 06:02:04,265-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] EVEN T_ID: VDS_ACTIVATE(16), Activation of host lago-upgrade-from-release-suite-master-host-0 initiated by admin@internal-authz. 2019-05-07 06:02:04,266-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-2) [70840477] Lock freed to object 'Engine Lock:{exclusiveLocks='[38e1379b-c3b6-4a2e-91df-d1f346e414a9=VDS]', sharedLocks=''}' 2019-05-07 06:02:04,484-04 ERROR [org.ovirt.engine.core.bll.hostdeploy.HostUpgradeCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-96) [05592db2-f859-487b-b779-4b32 eec5bab3] Host 'lago-upgrade-from-release-suite-master-host-0' failed to move to maintenance mode. Upgrade process is terminated. I can see there was only one vm running: drwxrwxr-x. 2 dron dron 1024 May 7 11:49 qemu [dron@dron post-004_basic_sanity.py]$ ls -l lago-upgrade-from-release-suite-master-host-0/_var_log/libvirt/qemu/ total 6 -rw-rw-r--. 1 dron dron 4466 May 7 10:12 vm-with-iface.log and I can see that there was an attempt to terminate it with an error that it does not exist: stroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0', gracefully='false', reason='', ig noreNoVm='false'}), log id: 24278e9b 2019-05-07 06:01:41,082-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] START, DestroyVDSComma nd(HostName = lago-upgrade-from-release-suite-master-host-0, DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', vmId='dfbd75e2-a9cb-4fca-8788-a169 54db4abf', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 78bba2f8 2019-05-07 06:01:42,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (default task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVDSComm and, return: , log id: 78bba2f8 2019-05-07 06:01:42,090-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (default task-1) [105f7555-517b-4bf9-b86e-6eb42375de20] FINISH, DestroyVmVDSCommand, r eturn: , log id: 24278e9b 2019-05-07 06:01:42,094-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-4) [] VM 'dfbd75e2-a9cb-4fca-8788-a16954db4abf' was reported as Down on VDS '38e1379b-c3b6-4a2e-91df-d1f346e414a9'(lago-upgrade-from-release-suite-master-host-0) 2019-05-07 06:01:42,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-4) [] START, DestroyVDSCommand(HostName = lago-upgrade- from-release-suite-master-host-0, DestroyVmVDSCommandParameters:{hostId='38e1379b-c3b6-4a2e-91df-d1f346e414a9', vmId='dfbd75e2-a9cb-4fca-8788-a16954db4abf', secondsToWait='0 ', gracefully='false', reason='', ignoreNoVm='true'}), log id: 1dbd31eb 2019-05-07 06:01:42,114-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-4) [] Failed to destroy VM 'dfbd75e2-a9cb-4fca-8788-a16 954db4abf' because VM does not exist, ignoring

3 10

manipulating VM configuration via Python SDK disables serial console
by Denis Karpov 10 May '19

10 May '19

Hello, I have hit what seems to be a bug in oVIrt Python SDK, when deploying OpenStack 14 (Rocky) in configuration virtualized controllers, oVirt VMs; Ironic oVirt driver is used in this case, which uses oVirt Python SDK to talk to manipulate VMs. The driver seem to work in such a way that at certain stages it retrieves a VM configuration and updates it (e.g. changes the boot device). However during such update some other changes in configuration of the VM happen: e.g. if the VM had VirtIO serial console enabled, after the update that serial console becomes disabled. Perhaps when a VMs config is updated some XML fields of its configuration are not written back as they should be? We have noticed that since the OpenStack deployment fails on controllers if it cannot output to the serial console. I would appreciate any help/workaround for this issue. Thanks, Denis

2 2

SATA interface?
by Hetz Ben Hamo 09 May '19

09 May '19

Hi, It looks like the Q35 chipset has SATA support for CDROM, but not for virtual hard disk. In many OS's/Appliances - there is no way to use virtio disks, and I can either use the extremly slow IDE with 440FX or nothing else. on Virt-Manager though, you can choose a SATA hard disk. So, is it possible to add the SATA support back for hard drives please?

3 3

manipulating VM configuration via Python SDK disables serial console
by Denis Karpov 08 May '19

08 May '19

Hello, I have hit what seems to be a bug in oVIrt Python SDK, when deploying OpenStack 14 (Rocky) in configuration virtualized controllers, oVirt VMs; Ironic oVirt driver is used in this case, which uses oVirt Python SDK to talk to manipulate VMs. The driver seem to work in such a way that at certain stages it retrieves a VM configuration and updates it (e.g. changes the boot device). However during such update some other changes in configuration of the VM happen: e.g. if the VM had VirtIO serial console enabled, after the update that serial console becomes disabled. Perhaps when a VMs config is updated some XML fields of its configuration are not written back as they should be? We have noticed that since the OpenStack deployment fails on controllers if it cannot output to the serial console. I would appreciate any help/workaround for this issue. Thanks, Denis

1 0

Fedora 29 support?
by Nir Soffer 08 May '19

08 May '19

Fedora 28 will be EOL in few month, and at this point people are migration to Fedora 29. I tried yesterday to build engine development setup on Fedora 29 VM, and found that it does not work: - some packages are missing on engine side - after hacking the ovirt.repo files, pointing to "28" instead of "$releasever", I got the packages... - but then adding a host fail with "Python is missing" on the host, which is working Fedora 28 host used with engine before. What are our plans for supporting Fedora 29 on engine and host? I hope that the work we did for Fedora 28 will make it easier to port to Fedora 29. Nir

3 5

OST broken - looks like https://gerrit.ovirt.org/c/99717 is the reason
by Nir Soffer 07 May '19

07 May '19

Investigating unrelated OST failue with: https://gerrit.ovirt.org/c/99293 I found that This patch fail in OST: https://gerrit.ovirt.org/c/99717 Failed build: https://jenkins.ovirt.org/job/ovirt-system-tests_manual/4630/ 2019-05-02 15:46:07,258::ssh.py::ssh::96::lago.ssh::DEBUG::Command 650bdf66 on lago-basic-suite-master-host-0 errors: + yum -y install ovirt-host Error: Package: vdsm-common-4.40.0-202.git9757d7c.el7.noarch (alocalsync) Requires: python2-enum34 Dan, can you take a look at this? Nir

4 5

Jenkins is very slow now - can someone restart it?
by Nir Soffer 05 May '19

05 May '19

$ time curl -s https://jenkins.ovirt.org/job/ovirt-system-tests_manual/ >/dev/null real 0m27.380s user 0m0.038s sys 0m0.028s For reference: $ time curl -s https://travis-ci.org/oVirt/vdsm/builds >/dev/null real 0m0.614s user 0m0.027s sys 0m0.015s Can someone restart jenkins?

3 4

ovirt-engine has been tagged (ovirt-engine-4.3.3.7)
by Tal Nisan 05 May '19

05 May '19

2 1

Jenkins restart
by Evgheni Dereveanchin 03 May '19

03 May '19

Hi everyone, We received reports that Jenkins is performing slowly so I'm going to restart it now to to apply some updates. During the outage no new builds will be started. I will announce once it is back up and running. -- Regards, Evgheni Dereveanchin

1 1

Thin provisioning??
by Hetz Ben Hamo 02 May '19

02 May '19

Hi, I've migrated few Linux VM's from virt-manager to oVirt. Other than the famous bug that it insist to have an ISO domain - it worked well. My problem is simple: on virt-manager, I created the virtual disks as thin provisioning. ls -l shows the full size of the QCOW2 file and du -hs shows the actual size. So far, so good. After migrating those VM's to oVirt, the disks appear both in the web UI and in the meta file as "Thin Provisioning" and "sparse" - however, digging into the directory which holds the virtual disk and running du -hs - shows it's a full size disk. Why is this happening? (the storage on the oVirt is NFS)

2 5