oVirt 4.5 linux guest vm with host device added to it fails to start
Hello I have a RHEL 8.6 based hypervisor with a Mellanox ConnectX-5 IB card installed with SRIOV enabled. The host device I am assigning is pci_0000_af_00_2. The card is working as I can talk to other infiniband interfaces on other servers. Below is the output of lspci. 3b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 3b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] The linux vm is configured as Q35 Chipset with UEFI, 16 cpus, numa enabled, and cpu pinning enabled. OS is RHEL 7.9. As soon as I start the vm, I get an immediate error message stating "Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable." If I remove the host device from the vm config, then it starts and runs fine. This setup was working just fine on RHEL8.4 and oVirt 4.4.7 using the proper driver for RHEL 8.4. Here is the engine.log after I press the run button. 2022-06-10 11:22:10,506-05 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[de54b903-7204-4966-95a3-05f64ed17f68=VM]', sharedLocks=''}' 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='de54b903-7204-4966-95a3-05f64ed17f68'}), log id: 6faf22a5 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6faf22a5 2022-06-10 11:22:10,560-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) [] Candidate host 'rvsh002' ('f68352c2-6ddc-44ae-a19b-9262e92327f8') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'HostDevice' (correlation id: null) 2022-06-10 11:22:10,569-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM ws006 due to a failed validation: [Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable.] (User: admin@internal-authz). 2022-06-10 11:22:10,569-05 WARN [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [] Validation of action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName rvsh002,$filterName HostDevice,VAR__DETAIL__HOST_DEVICE_UNAVAILABLE,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL 2022-06-10 11:22:10,570-0 There was nothing in the vdsm.log on the hypervisor related to this issue that I could see after hitting the run button. Thanks Don
THis is for version oVirt 4.5.0.8-1. Sorry left out the exact release. Don On Fri, Jun 10, 2022 at 11:41 AM Don Dupuis <dondster@gmail.com> wrote:
Hello I have a RHEL 8.6 based hypervisor with a Mellanox ConnectX-5 IB card installed with SRIOV enabled. The host device I am assigning is pci_0000_af_00_2. The card is working as I can talk to other infiniband interfaces on other servers. Below is the output of lspci. 3b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 3b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
The linux vm is configured as Q35 Chipset with UEFI, 16 cpus, numa enabled, and cpu pinning enabled. OS is RHEL 7.9. As soon as I start the vm, I get an immediate error message stating "Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable." If I remove the host device from the vm config, then it starts and runs fine. This setup was working just fine on RHEL8.4 and oVirt 4.4.7 using the proper driver for RHEL 8.4.
Here is the engine.log after I press the run button. 2022-06-10 11:22:10,506-05 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[de54b903-7204-4966-95a3-05f64ed17f68=VM]', sharedLocks=''}' 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='de54b903-7204-4966-95a3-05f64ed17f68'}), log id: 6faf22a5 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6faf22a5 2022-06-10 11:22:10,560-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) [] Candidate host 'rvsh002' ('f68352c2-6ddc-44ae-a19b-9262e92327f8') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'HostDevice' (correlation id: null) 2022-06-10 11:22:10,569-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM ws006 due to a failed validation: [Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable.] (User: admin@internal-authz). 2022-06-10 11:22:10,569-05 WARN [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [] Validation of action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName rvsh002,$filterName HostDevice,VAR__DETAIL__HOST_DEVICE_UNAVAILABLE,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL 2022-06-10 11:22:10,570-0
There was nothing in the vdsm.log on the hypervisor related to this issue that I could see after hitting the run button.
Thanks Don
Hello Anyone have any ideas? Don On Fri, Jun 10, 2022 at 11:45 AM Don Dupuis <dondster@gmail.com> wrote:
THis is for version oVirt 4.5.0.8-1. Sorry left out the exact release.
Don
On Fri, Jun 10, 2022 at 11:41 AM Don Dupuis <dondster@gmail.com> wrote:
Hello I have a RHEL 8.6 based hypervisor with a Mellanox ConnectX-5 IB card installed with SRIOV enabled. The host device I am assigning is pci_0000_af_00_2. The card is working as I can talk to other infiniband interfaces on other servers. Below is the output of lspci. 3b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 3b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
The linux vm is configured as Q35 Chipset with UEFI, 16 cpus, numa enabled, and cpu pinning enabled. OS is RHEL 7.9. As soon as I start the vm, I get an immediate error message stating "Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable." If I remove the host device from the vm config, then it starts and runs fine. This setup was working just fine on RHEL8.4 and oVirt 4.4.7 using the proper driver for RHEL 8.4.
Here is the engine.log after I press the run button. 2022-06-10 11:22:10,506-05 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[de54b903-7204-4966-95a3-05f64ed17f68=VM]', sharedLocks=''}' 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='de54b903-7204-4966-95a3-05f64ed17f68'}), log id: 6faf22a5 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6faf22a5 2022-06-10 11:22:10,560-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) [] Candidate host 'rvsh002' ('f68352c2-6ddc-44ae-a19b-9262e92327f8') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'HostDevice' (correlation id: null) 2022-06-10 11:22:10,569-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM ws006 due to a failed validation: [Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable.] (User: admin@internal-authz). 2022-06-10 11:22:10,569-05 WARN [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [] Validation of action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName rvsh002,$filterName HostDevice,VAR__DETAIL__HOST_DEVICE_UNAVAILABLE,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL 2022-06-10 11:22:10,570-0
There was nothing in the vdsm.log on the hypervisor related to this issue that I could see after hitting the run button.
Thanks Don
Don Dupuis <dondster@gmail.com> writes:
Hello Anyone have any ideas?
Don
On Fri, Jun 10, 2022 at 11:45 AM Don Dupuis <dondster@gmail.com> wrote:
THis is for version oVirt 4.5.0.8-1. Sorry left out the exact release.
Don
On Fri, Jun 10, 2022 at 11:41 AM Don Dupuis <dondster@gmail.com> wrote:
Hello I have a RHEL 8.6 based hypervisor with a Mellanox ConnectX-5 IB card installed with SRIOV enabled. The host device I am assigning is pci_0000_af_00_2. The card is working as I can talk to other infiniband interfaces on other servers. Below is the output of lspci. 3b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 3b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
The linux vm is configured as Q35 Chipset with UEFI, 16 cpus, numa enabled, and cpu pinning enabled. OS is RHEL 7.9. As soon as I start the vm, I get an immediate error message stating "Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable." If I remove the host device from the vm config, then it starts and runs fine. This setup was working just fine on RHEL8.4 and oVirt 4.4.7 using the proper driver for RHEL 8.4.
Engine apparently cannot find a host with enough CPUs and free memory, matching the NUMA and CPU pinning configurations, and having the given host device available. According to the log, rvsh002 doesn't have the host device, other hosts apparently don't satisfy some of the other conditions. Also, isn't the VM pinned to some hosts? Maybe someone could provide a better advice, but if you think there is a host satisfying all the conditions, you can try to start the VM there with "Run Once" and see if Engine provides a reason why it cannot be started there.
Here is the engine.log after I press the run button. 2022-06-10 11:22:10,506-05 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[de54b903-7204-4966-95a3-05f64ed17f68=VM]', sharedLocks=''}' 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='de54b903-7204-4966-95a3-05f64ed17f68'}), log id: 6faf22a5 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6faf22a5 2022-06-10 11:22:10,560-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) [] Candidate host 'rvsh002' ('f68352c2-6ddc-44ae-a19b-9262e92327f8') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'HostDevice' (correlation id: null) 2022-06-10 11:22:10,569-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM ws006 due to a failed validation: [Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable.] (User: admin@internal-authz). 2022-06-10 11:22:10,569-05 WARN [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [] Validation of action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName rvsh002,$filterName HostDevice,VAR__DETAIL__HOST_DEVICE_UNAVAILABLE,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL 2022-06-10 11:22:10,570-0
There was nothing in the vdsm.log on the hypervisor related to this issue that I could see after hitting the run button.
Engine couldn't find a matching host so the VM is not attempted to start anywhere.
Thanks Don
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GZV47I6D6C4EVG...
Don Dupuis <dondster@gmail.com> writes:
Hello Anyone have any ideas?
Don
On Fri, Jun 10, 2022 at 11:45 AM Don Dupuis <dondster@gmail.com> wrote:
THis is for version oVirt 4.5.0.8-1. Sorry left out the exact release.
Don
On Fri, Jun 10, 2022 at 11:41 AM Don Dupuis <dondster@gmail.com> wrote:
Hello I have a RHEL 8.6 based hypervisor with a Mellanox ConnectX-5 IB card installed with SRIOV enabled. The host device I am assigning is pci_0000_af_00_2. The card is working as I can talk to other infiniband interfaces on other servers. Below is the output of lspci. 3b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 3b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
The linux vm is configured as Q35 Chipset with UEFI, 16 cpus, numa enabled, and cpu pinning enabled. OS is RHEL 7.9. As soon as I start the vm, I get an immediate error message stating "Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable." If I remove the host device from the vm config, then it starts and runs fine. This setup was working just fine on RHEL8.4 and oVirt 4.4.7 using the proper driver for RHEL 8.4.
Engine apparently cannot find a host with enough CPUs and free memory, matching the NUMA and CPU pinning configurations, and having the given host device available. According to the log, rvsh002 doesn't have the host device, other hosts apparently don't satisfy some of the other conditions. Also, isn't the VM pinned to some hosts? Maybe someone could provide a better advice, but if you think there is a host satisfying all the conditions, you can try to start the VM there with "Run Once" and see if Engine provides a reason why it cannot be started there.
Here is the engine.log after I press the run button. 2022-06-10 11:22:10,506-05 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[de54b903-7204-4966-95a3-05f64ed17f68=VM]', sharedLocks=''}' 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='de54b903-7204-4966-95a3-05f64ed17f68'}), log id: 6faf22a5 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6faf22a5 2022-06-10 11:22:10,560-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) [] Candidate host 'rvsh002' ('f68352c2-6ddc-44ae-a19b-9262e92327f8') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'HostDevice' (correlation id: null) 2022-06-10 11:22:10,569-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM ws006 due to a failed validation: [Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable.] (User: admin@internal-authz). 2022-06-10 11:22:10,569-05 WARN [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [] Validation of action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName rvsh002,$filterName HostDevice,VAR__DETAIL__HOST_DEVICE_UNAVAILABLE,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL 2022-06-10 11:22:10,570-0
There was nothing in the vdsm.log on the hypervisor related to this issue that I could see after hitting the run button.
Engine couldn't find a matching host so the VM is not attempted to start anywhere.
Thanks Don
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GZV47I6D6C4EVG...
participants (2)
-
Don Dupuis -
Milan Zamazal