September 2022 - Users - oVirt List Archives

VM Down With "Bad Volume Specification"
by Clint Boggio 21 Sep '22

21 Sep '22

I had occasion to shutdown a VM for the purpose of adding RAM and processor to it and the VM willnot boot back up. I'm seeing "VM Issabel_PBX is down with error. Exit message: Bad volume specification {'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'serial': '6af66318-e6f8-45d7-8b4e-2183faf0a917', 'index': 0, 'iface': 'scsi', 'apparentsize': '5706743808', 'specParams': {}, 'cache': 'none', 'imageID': '6af66318-e6f8-45d7-8b4e-2183faf0a917', 'truesize': '5950070784', 'type': 'disk', 'domainID': '24c4dc1b-c843-4ae2-963f-9d0548305192', 'reqsize': '0', 'format': 'cow', 'poolID': '31fdd642-6b06-11ea-a4c4-00163e333bd2', 'device': 'disk', 'path': '/rhev/data-center/31fdd642-6b06-11ea-a4c4-00163e333bd2/24c4dc1b-c843-4ae2-963f-9d0548305192/images/6af66318-e6f8-45d7-8b4e-2183faf0a917/576b2761-a5bc-427b-95a9-0594447f0705', 'propagateErrors': 'off', 'name': 'sda', 'bootOrder': '1', 'volumeID': '576b2761-a5bc-427b-95a9-0594447f0705', 'diskType': 'file', 'a lias': 'ua-6af66318-e6f8-45d7-8b4e-2183faf0a917', 'discard': False}." in the log. I tried to move the VMs disk from one gluster datastore to another to see if the problem would clear and now the disk is locked and the move is stuck at %10. In the engine logs I have "2022-09-19 12:48:25,614-05 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task '4b96a8e1-ab65-4d1c-97dd-e985ab7816c6' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2022-09-19 12:48:25,614-05 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] Task id '4b96a8e1-ab65-4d1c-97dd-e985ab7816c6' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2022-09-19 12:48:25,631-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] EVENT_ID: TASK_CLEARING_ASYNC_TASK(9,501), Clearing asynchronous task Unknown that started at Tue Jul 12 12:19:21 CDT 2022 2022-09-19 12:48:25,631-05 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] Cleaning zombie tasks: Clearing async task 'Unknown' that started at 'Tue Jul 12 12:19:21 CDT 2022' since it reached a timeout of 3000 minutes 2022-09-19 12:48:25,631-05 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '4b96a8e1-ab65-4d1c-97dd-e985ab7816c6' 2022-09-19 12:48:25,632-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='31fdd642-6b06-11ea-a4c4-00163e333bd2', ignoreFailoverLimit='false', taskId='4b96a8e1-ab65-4d1c-97dd-e985ab7816c6'}), log id: 40d6d67f 2022-09-19 12:48:25,633-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] START, HSMClearTaskVDSCommand(HostName = hprvsr00.locacore.com, HSMTaskGuidBaseVDSCommandParameters:{hostId='6c910725-fb42-4a64-b614-2a29bf0800e2', taskId='4b96a8e1-ab65-4d1c-97dd-e985ab7816c6'}), log id: 22c28060 2022-09-19 12:48:25,638-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] FINISH, HSMClearTaskVDSCommand, return: , log id: 22c28060 2022-09-19 12:48:25,639-05 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] FINISH, SPMClearTaskVDSCommand, return: , log id: 40d6d67f 2022-09-19 12:48:25,639-05 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task '4b96a8e1-ab65-4d1c-97dd-e985ab7816c6' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2022-09-19 12:48:25,876-05 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-98) [149db168-34c0-4814-869e-1ca0fdbde768] Command 'CopyImageGroupWithData' (id: '9e73adde-e485-461a-b349-7fd814890aa6') waiting on child command id: '5b4d0a00-7e40-4016-bf5c-0db013e22983' type:'CopyImageGroupVolumesData' to complete 2022-09-19 12:48:26,878-05 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-84) [149db168-34c0-4814-869e-1ca0fdbde768] Command 'CopyImageGroupVolumesData' (id: '5b4d0a00-7e40-4016-bf5c-0db013e22983') waiting on child command id: '1b496e55-9407-4fd7-a2f2-bb70bf4e7aa0' type:'CopyData' to complete 2022-09-19 12:48:27,886-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-89) [149db168-34c0-4814-869e-1ca0fdbde768] START, GetHostJobsVDSCommand(HostName = hprvsr00.locacore.com, GetHostJobsVDSCommandParameters:{hostId='6c910725-fb42-4a64-b614-2a29bf0800e2', type='storage', jobIds='[06208564-b66d-4947-a96f-4d163ef2fbe0]'}), log id: 75ca6198 2022-09-19 12:48:27,894-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHostJobsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-89) [149db168-34c0-4814-869e-1ca0fdbde768] FINISH, GetHostJobsVDSCommand, return: {06208564-b66d-4947-a96f-4d163ef2fbe0=HostJobInfo:{id='06208564-b66d-4947-a96f-4d163ef2fbe0', type='storage', description='copy_data', status='running', progress='null', error='null'}}, log id: 75ca6198 2022-09-19 12:48:27,902-05 INFO [org.ovirt.engine.core.bll.StorageJobCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-89) [149db168-34c0-4814-869e-1ca0fdbde768] Command CopyData id: '1b496e55-9407-4fd7-a2f2-bb70bf4e7aa0': waiting for job '06208564-b66d-4947-a96f-4d163ef2fbe0' on host 'hprvsr00.locacore.com' (id: '6c910725-fb42-4a64-b614-2a29bf0800e2') to complete 2022-09-19 12:48:29,911-05 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [149db168-34c0-4814-869e-1ca0fdbde768] Command 'MoveOrCopyDisk' (id: '2ec06560-b552-4418-abd7-e2945cd98c12') waiting on child command id: '80a2481e-707f-49bf-b469-33cd90c1a51c' type:'MoveImageGroup' to complete 2022-09-19 12:48:29,915-05 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-57) [149db168-34c0-4814-869e-1ca0fdbde768] Command 'MoveImageGroup' (id: '80a2481e-707f-49bf-b469-33cd90c1a51c') waiting on child command id: '9e73adde-e485-461a-b349-7fd814890aa6' type:'CopyImageGroupWithData' to complete" Any help would be appreciated as the client's PBX is currently down as a result.

2 4

Re: Self-hosted-engine timeout and recovering time
by Yedidyah Bar David 21 Sep '22

21 Sep '22

On Wed, Sep 21, 2022 at 12:22 AM Marcos Sungaila <marcos.sungaila(a)oracle.com> wrote: > > Hi all, > > I have a cluster running the 4.4.10 release with 6 KVM hosts and Self-Hosted-Engine. What storage? > I'm testing some network outage scenarios, and I faced strange behavior. I suppose you have redundancy in your network. It's important to clarify (for yourself, mainly) what exactly you test, what's important, what's expected, etc. > After disconnecting the KVM hosts hosting the SHE, there was a long timeout until switching the Self-Hosted-Engine to another host as expected. I suggest studying the ha-agent logs, /var/log/ovirt-hosted-engine-ha/agent.log. Much of the relevant code is in ovirt_hosted_engine_ha/agent/states.py (in the git repo, or under /usr/lib/python3.6/site-packages/ on your machine). > Also, there took a relatively long time to take over the HA VMs from the failing server. That's a separate issue, about which I personally know very little. You might want to start a separate thread about it. I do know, though, that if you keep the storage connected, the host might be able to keep updating VM leases on the storage. See e.g.: https://www.ovirt.org/develop/release-management/features/storage/vm-leases… I didn't check the admin guide, but I suppose it has some material about HA VMs. > Is there a configuration where I can reduce the SHE timeout to make this recover process faster? IIRC there is nothing user-configurable. You can see most relevant constants in ovirt_hosted_engine_ha/agent/constants.py{,.in}. Nothing stops you from changing them, but please note that this is somewhat risky, and I strongly suggest to do very careful testing with your new settings. It might make sense to try to methodically go through all the possible state changes in the above state machine. The general assumption is that network and storage, for critical setups, are redundant, and that the engine itself is not considered critical, in the sense that if it's dead, all your VMs are still alive. And also, that it's more important to not corrupt VM disk images (e.g. by starting the VM concurrently on two hosts) than to keep the VM alive. Best regards, -- Didi

2 1

all active domains with status unknown in old 4.3 cluster
by Jorick Astrego 21 Sep '22

21 Sep '22

Hi, Currently I'm debugging a client's ovirt 4.3 cluster. I was adding two new gluster domains and got a timeout "VDSM command AttachStorageDomainVDS failed: Resource timeout: ()" and "Failed to attach Storage Domain *** to Data Center **". Then I had to restart ovirt-engine and now all the domains including NFS domains have status "unknown" and I see "VDSM command GetStoragePoolInfoVDS failed: Resource timeout: ()" in the events. Anyone fixed this before or have any tips? Met vriendelijke groet, With kind regards, Jorick Astrego Netbulae Virtualization Experts ---------------- Tel: 053 20 30 270 info(a)netbulae.eu Staalsteden 4-3A KvK 08198180 Fax: 053 20 30 271 www.netbulae.eu 7547 TA Enschede BTW NL821234584B01 ----------------

1 0

Snapshot task stuck at oVirt 4.4.8
by nicolas＠devels.es 21 Sep '22

21 Sep '22

Hi, We're running oVirt 4.4.8 and one of our users tried to create a snapshot on a VM. The snapshot task got stuck (not sure why) and since then a "locked" icon is being shown on the VM. We need to remove this VM, but since it has a pending task, we're unable. The ovirt-engine log shows hundreds of events like: [2022-09-20 09:23:09,286+01 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [2769dad5-3ec3-4c46-90a2-924746ea8d97] Command 'CreateSnapshotForVm' (id: '4fcb6ab7-2cd7-4a0c-be97-f6979be25bb9') waiting on child command id: 'cbb7a2c0-2111-4958-a55d-d48bf2d8591b' type:'CreateLiveSnapshotForVm' to complete An ovirt-engine restart didn't make any difference. Is there a way to remove this task manually, even changing something in the DB? Thanks.

4 4

oVirt Engine VM On Rocky Linux
by Matthew J Black 21 Sep '22

21 Sep '22

Hi Everybody (Hi Dr. Nick), Has anyone attempted to migrate the oVirt Engine VM over to Rocky Linux (v8.6), and if so, any "gotchas" we need to know about? Cheers Dulux-Oz

1 0

oVirt & (Ceph) iSCSI
by Matthew J Black 21 Sep '22

21 Sep '22

Hi Everybody (Hi Dr. Nick), So, next question in my on-going saga: *somewhere* in the documentation I read that when using oVirt with multiple iSCSI paths (in my case, multiple Ceph iSCSI Gateways) we need to set up DM Multipath. My question is: Is this still relevant information when using oVirt v4.5.2? Relevant link referred to by the oVirt Documentation: - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… Cheers Dulux-Oz

1 0

Self-hosted-engine timeout and recovering time
by Marcos Sungaila 20 Sep '22

20 Sep '22

Hi all, I have a cluster running the 4.4.10 release with 6 KVM hosts and Self-Hosted-Engine. I'm testing some network outage scenarios, and I faced strange behavior. After disconnecting the KVM hosts hosting the SHE, there was a long timeout until switching the Self-Hosted-Engine to another host as expected. Also, there took a relatively long time to take over the HA VMs from the failing server. Is there a configuration where I can reduce the SHE timeout to make this recover process faster? Regards, Marcos Sungaila

1 0

How do I migrate a running VM off unassigned host?
by David White 20 Sep '22

20 Sep '22

Ok, now that I'm able to (re)deploy ovirt to new hosts, I now need to migrate VMs that are running on hosts that are currently in an "unassigned" state in the cluser. This is the result of having moved the oVirt engine OUT of a hyperconverged environment onto its own stand-alone system, while simultaneously upgrading oVirt from v4.4 to the latest v4.5. See the following email threads: - https://lists.ovirt.org/archives/list/users@ovirt.org/thread/TZAUCM3GB5ERST… - https://lists.ovirt.org/archives/list/users@ovirt.org/thread/3IWXZ7VXM6CYYS… The oVirt engine knows about the VMs, and oVirt knows about the storage that those VMs are on. But the engine sees 2 of my hosts as "unassigned", and I've been unable to migrate the disks to new storage, nor live migrate a VM from an unassigned host, nor make a clone of an existing VM. Is there a way to recover from this scenario? I was thinking something along the lines of manually shutting down the VM on the unassigned host, and then somehow force the engine to bring the VM online again from a healthy host? Thanks, David Sent with Proton Mail secure email.

2 4

long time running backup (hanged in image finalizing state )
by Jirka Simon 19 Sep '22

19 Sep '22

Hello there. we have issue with backups on our cluster, one backup started 2 days ago and is is still in state finalizing. select * from vm_backups; backup_id | from_checkpoint_id | to_checkpoint_id | vm_id | phase | _create_date | host_id | des cription | _update_date | backup_type | snapshot_id | is_stopped --------------------------------------+--------------------+--------------------------------------+--------------------------------------+-------+----------------------------+---------+---- ---------+----------------------------+-------------+--------------------------------------+------------ b9c458e6-64e2-41c2-93b8-96761e71f82b | | 7a558f2a-57b6-432f-b5dd-85f5fb9dac8e | c3b2199f-35cc-41dc-8787-835e945217d2 | Ready | 2022-09-17 00:44:56.877+02 | | | 2022-09-17 00:45:19.057+02 | hybrid | 0c6ebd56-dcfe-46a8-91cc-327cc94e9773 | f (1 row) and if I check imagetransfer table, I see bytes_sent = bytes_total. engine=# select it.disk_id,bd.disk_alias,it.last_updated, it.bytes_sent, it.bytes_total from image_transfers as it , base_disks as bd where it.disk_id = bd.disk_id; disk_id | disk_alias | last_updated | bytes_sent | bytes_total --------------------------------------+-------------------------------------------------------+----------------------------+--------------+-------------- 950279ef-485c-400e-ba66-a3f545618de5 | log1.util.prod.hq.sldev.cz_log1.util.prod.hq.sldev.cz | 2022-09-17 01:43:09.229+02 | 214748364800 | 214748364800 there is no error in logs if i use /usr/share/ovirt-engine/setup/dbutils/unlock_entity.sh -t all -qc there is no record in any part. I can clean these record from DB to fix it but it will happen again in few days. vdsm.x86_64 4.50.2.2-1.el8 ovirt-engine.noarch 4.5.2.4-1.el8 is there anything i can check to find reason of this ? Thank you Jirka

2 8

Unable to deploy to new host
by David White 19 Sep '22

19 Sep '22

I currently have a self-hosted engine that was restored from a backup of an engine that was originally in a hyperconverged state. (See https://lists.ovirt.org/archives/list/users@ovirt.org/message/APQ3XBUM34TG7…) This was also an upgrade from ovirt 4.4 to ovirt 4.5. There were 4 hosts in this cluster. Unfortunately, 2 of them are completely in an "Unassigned" state right now, and I don't know why. The VMs on those hosts are working fine, but I have no way to move the VMs or manage them. More to the point of this email: I'm trying to re-deploy onto a 3rd host. I did a fresh install of Rocky Linux 8, and followed the instructions at https://ovirt.org/download/ and at https://ovirt.org/download/install_on_rhel.html, including the part there that is specific to Rocky. After installing the centos-release-ovirt45 package, I then logged into the oVirt engine web UI, and went to Compute -> Hosts -> New, and have tried (and failed) many times to install / deploy to this new host. The last error in the host deploy log is the following: 2022-09-18 21:29:39 EDT - { "uuid" : "94b93e6a-5410-4d26-b058-d7d1db0a151e", "counter" : 404, "stdout" : "fatal: [cha2-storage.mgt.example.com] FAILED! => {\"msg\": \"The conditional check 'cluster_switch == \\\"ovs\\\" or (ovn_central is defined and ovn_central | ipaddr)' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller\\n\\nThe error appears to be in '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be:\\n\\n- block:\\n - name: Install ovs\\n ^ here\\n\"}", "start_line" : 405, "end_line" : 406, "runner_ident" : "e2cbd38d-64fa-4ecd-82c6-114420ea14a4", "event" : "runner_on_failed", "pid" : 65899, "created" : "2022-09-19T01:29:38.983937", "parent_uuid" : "02113221-f1b3-920f-8bd4-00000000003d", "event_data" : { "playbook" : "ovirt-host-deploy.yml", "playbook_uuid" : "73a6e8f1-3836-49e1-82fd-5367b0bf4e90", "play" : "all", "play_uuid" : "02113221-f1b3-920f-8bd4-000000000006", "play_pattern" : "all", "task" : "Install ovs", "task_uuid" : "02113221-f1b3-920f-8bd4-00000000003d", "task_action" : "package", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml:3", "role" : "ovirt-provider-ovn-driver", "host" : "cha2-storage.mgt.example.com", "remote_addr" : "cha2-storage.mgt.example.com", "res" : { "msg" : "The conditional check 'cluster_switch == \"ovs\" or (ovn_central is defined and ovn_central | ipaddr)' failed. The error was: The ipaddr filter requires python's netaddr be installed on the ansible controller\n\nThe error appears to be in '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml': line 3, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n- block:\n - name: Install ovs\n ^ here\n", "_ansible_no_log" : false }, "start" : "2022-09-19T01:29:38.919334", "end" : "2022-09-19T01:29:38.983680", "duration" : 0.064346, "ignore_errors" : null, "event_loop" : null, "uuid" : "94b93e6a-5410-4d26-b058-d7d1db0a151e" } } On the engine, I have verified that netaddr is installed. And just for kicks, I've installed as many different versions as I can find: [root@ovirt-engine1 host-deploy]# rpm -qa | grep netaddrpython38-netaddr-0.7.19-8.1.1.el8.noarch python2-netaddr-0.7.19-8.1.1.el8.noarch python3-netaddr-0.7.19-8.1.1.el8.noarch The engine is based on CentOS Stream 8 (when I moved the engine out of the hyperconverged environment, my goal was to keep things as close to the original environment as possible) [root@ovirt-engine1 host-deploy]# cat /etc/redhat-release CentOS Stream release 8 The engine is fully up-to-date: [root@ovirt-engine1 host-deploy]# uname -a Linux ovirt-engine1.mgt.barredowlweb.com 4.18.0-408.el8.x86_64 #1 SMP Mon Jul 18 17:42:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux And the engine has the following repos: [root@ovirt-engine1 host-deploy]# yum repolistrepo id repo name appstream CentOS Stream 8 - AppStream baseos CentOS Stream 8 - BaseOS centos-ceph-pacific CentOS-8-stream - Ceph Pacific centos-gluster10 CentOS-8-stream - Gluster 10 centos-nfv-openvswitch CentOS-8 - NFV OpenvSwitch centos-opstools CentOS-OpsTools - collectd centos-ovirt45 CentOS Stream 8 - oVirt 4.5 extras CentOS Stream 8 - Extras extras-common CentOS Stream 8 - Extras common packages ovirt-45-centos-stream-openstack-yoga CentOS Stream 8 - oVirt 4.5 - OpenStack Yoga Repository ovirt-45-upstream oVirt upstream for CentOS Stream 8 - oVirt 4.5 powertools CentOS Stream 8 - PowerTools Why does deploying to this new Rocky host keep failing? Sent with Proton Mail secure email.

2 5