June 2020 - Users - oVirt List Archives

Unable to install oVirt on RHEL7.5
by SS00514758＠techmahindra.com 15 Jan '26

15 Jan '26

Hi All, I am unable to install oVirt on RHEL7.5, to install it I am taking reference of below link, https://www.ovirt.org/documentation/install-guide/chap-Installing_oVirt.html But though it is not working for me, couple of dependencies is not getting installed, and because of this I am not able to run the ovirt-engine, below are the depencies packages that unable to install, Error: Package: collectd-write_http-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) Requires: collectd(x86-64) = 5.8.0-6.1.el7 Removing: collectd-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-6.1.el7 Updated By: collectd-5.8.1-1.el7.x86_64 (epel) collectd(x86-64) = 5.8.1-1.el7 Available: collectd-5.7.2-1.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-1.el7 Available: collectd-5.7.2-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-3.el7 Available: collectd-5.8.0-2.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-2.el7 Available: collectd-5.8.0-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-3.el7 Available: collectd-5.8.0-5.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-5.el7 Help me to install this. Looking forward to resolve this issue. Regards Sumit Sahay

8 10

SPM and Task error ...
by Enrico 22 Nov '24

22 Nov '24

Hi all, my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7, ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel). I need to stop one of the hypervisors for maintenance but this system is the storage pool manager. For this reason I decided to manually activate SPM in one of the other nodes but this operation is not successful. In the ovirt engine (engine.log) the error is this: 2019-07-25 12:39:16,744+02 INFO [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] Running command: ForceSelectSPMCommand internal: false. Entities affected : ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-07-25 12:39:16,745+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false'}), log id: 37bf4639 2019-07-25 12:39:16,747+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', ignoreStopFailed='false'}), log id: 2522686f 2019-07-25 12:39:16,749+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopVDSCommand(HostName = infn-vm05.management, SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log id: 1810fd8b 2019-07-25 12:39:16,758+02 *ERROR* [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] SpmStopVDSCommand::Not stopping SPM on vds 'infn-vm05.management', pool id '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status 'running'' 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopVDSCommand, log id: 1810fd8b 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, ResetIrsVDSCommand, log id: 2522686f 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 2019-07-25 12:39:16,760+02 *ERROR* [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed to force select infn-vm07.management as the SPM due to a failure to stop the current SPM. while in the hypervisor (SPM) vdsm.log: 2019-07-25 12:39:16,744+02 INFO [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] Running command: ForceSelectSPMCommand internal: false. Entities affected : ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-07-25 12:39:16,745+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false'}), log id: 37bf4639 2019-07-25 12:39:16,747+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', ignoreStopFailed='false'}), log id: 2522686f 2019-07-25 12:39:16,749+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopVDSCommand(HostName = infn-vm05.management, SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log id: 1810fd8b 2019-07-25 12:39:16,758+02 *ERROR* [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] SpmStopVDSCommand::Not stopping SPM on vds 'infn-vm05.management', pool id '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status 'running'' 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopVDSCommand, log id: 1810fd8b 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, ResetIrsVDSCommand, log id: 2522686f 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 2019-07-25 12:39:16,760+02 *ERROR* [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed to force select infn-vm07.management as the SPM due to a failure to stop the current SPM. 2019-07-25 12:39:18,660+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,660+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,750+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,750+02 *ERROR* [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2019-07-25 12:39:18,750+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,751+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 34ae2b2f 2019-07-25 12:39:18,752+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: d3a78ad 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: d3a78ad 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: 34ae2b2f 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 42de0c2b 2019-07-25 12:39:18,759+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 4895c79c 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: 4895c79c 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: 42de0c2b 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Cleaning zombie tasks: Clearing async task 'Unknown' that started at 'Fri May 03 14:48:50 CEST 2019' 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,765+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: da77af2 2019-07-25 12:39:18,766+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 530694fb 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: 530694fb 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: da77af2 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned there is some relation between this error and a task that has remained hanging, from SPM server: # vdsm-client Task getInfo taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e { "verb": "prepareMerge", "id": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e" } # vdsm-client Task getStatus taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e { "message": "running job 1 of 1", "code": 0, "taskID": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e", "taskResult": "", "taskState": "running" } How can I solve this problem ? Thanks a lot for your help !! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________

5 15

Changing disk QoS causes segfault with IO-Threads enabled (oVirt 4.3.0.4-1.el7)
by jloh＠squiz.net 23 Apr '24

23 Apr '24

We recently upgraded to 4.3.0 and have found that when changing disk QoS settings on VMs whilst IO-Threads is enabled causes them to segfault and the VM to reboot. We've been able to replicate this across several VMs. VMs with IO-Threads disabled/turned off do not segfault when changing the QoS. Mar 1 11:49:06 srvXX kernel: IO iothread1[30468]: segfault at fffffffffffffff8 ip 0000557649f2bd24 sp 00007f80de832f60 error 5 in qemu-kvm[5576498dd000+a03000] Mar 1 11:49:06 srvXX abrt-hook-ccpp: invalid number 'iothread1' Mar 1 11:49:11 srvXX libvirtd: 2019-03-01 00:49:11.116+0000: 13365: error : qemuMonitorIORead:609 : Unable to read from monitor: Connection reset by peer Happy to supply some more logs to someone if they'll help but just wondering whether anyone else has experienced this or knows of a current fix other than turning io-threads off. Cheers.

4 3

Re: Failed to synchronize networks of Provider ovirt-provider-ovn
by Mail SET Inc. Group 27 Oct '23

27 Oct '23

Yes, i use same manual to change WebUI SSL. ovirt-ca-file= is a same SSL file which use WebUI. Yes, i restart ovirt-provider-ovn, i restart engine, i restart all what i can restart. Nothing... > 12 сент. 2018 г., в 16:11, Dominik Holler <dholler(a)redhat.com> написал(а): > > On Wed, 12 Sep 2018 14:23:54 +0300 > "Mail SET Inc. Group" <mail(a)set-pro.net> wrote: > >> Ok! > > Not exactly, please use users(a)ovirt.org for such questions. > Other should benefit from this questions, too. > Please write the next mail to users(a)ovirt.org and keep me in CC. > >> What i did: >> >> 1) install oVirt «from box» (4.2.5.2-1.el7); >> 2) generate own ssl for my engine using my FreeIPA CA, Install it and > > What means "Install it"? You can use the doc from the following link > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/ht… > > Ensure that ovirt-ca-file= in > /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf > points to the correct file and ovirt-provider-ovn is restarted. > >> get tis issue; >> >> >> [root@engine ~]# tail -n 50 /var/log/ovirt-provider-ovn.log >> 2018-09-12 14:10:23,828 root [SSL: CERTIFICATE_VERIFY_FAILED] >> certificate verify failed (_ssl.c:579) Traceback (most recent call >> last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", >> line 133, in _handle_request method, path_parts, content >> File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", >> line 175, in handle_request return >> self.call_response_handler(handler, content, parameters) File >> "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in >> call_response_handler return response_handler(content, parameters) >> File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", >> line 62, in post_tokens user_password=user_password) File >> "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in >> create_token return auth.core.plugin.create_token(user_at_domain, >> user_password) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line >> 48, in create_token timeout=self._timeout()) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, >> in create_token username, password, engine_url, ca_file, timeout) >> File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line >> 91, in _get_sso_token timeout=timeout File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, >> in wrapper response = func(*args, **kwargs) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, >> in wrapper raise BadGateway(e) BadGateway: [SSL: >> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) >> >> >> [root@engine ~]# tail -n 20 /var/log/ovirt-engine/engine.log >> 2018-09-12 14:10:23,773+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock >> Acquired to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:10:23,778+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] >> Running command: SyncNetworkProviderCommand internal: true. >> 2018-09-12 14:10:23,836+03 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] >> Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:10:23,837+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock >> freed to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:14:12,477+03 INFO >> [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default >> task-6) [] User admin@internal successfully logged in with scopes: >> ovirt-app-admin ovirt-app-api ovirt-app-portal >> ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all >> ovirt-ext=token-info:authz-search >> ovirt-ext=token-info:public-authz-search >> ovirt-ext=token-info:validate ovirt-ext=token:password-access >> 2018-09-12 14:14:12,587+03 INFO >> [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default >> task-6) [1bf1b763] Running command: CreateUserSessionCommand >> internal: false. 2018-09-12 14:14:12,628+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-6) [1bf1b763] EVENT_ID: USER_VDC_LOGIN(30), User >> admin@internal-authz connecting from '10.0.3.61' using session >> 's8jAm7BUJGlicthm6yZBA3CUM8QpRdtwFaK3M/IppfhB3fHFB9gmNf0cAlbl1xIhcJ2WX+ww7e71Ri+MxJSsIg==' >> logged in. 2018-09-12 14:14:30,972+03 INFO >> [org.ovirt.engine.core.bll.provider.ImportProviderCertificateCommand] >> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] Running >> command: ImportProviderCertificateCommand internal: false. Entities >> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: >> SystemAction group CREATE_STORAGE_POOL with role type ADMIN >> 2018-09-12 14:14:30,982+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] EVENT_ID: >> PROVIDER_CERTIFICATE_IMPORTED(213), Certificate for provider >> ovirt-provider-ovn was imported. (User: admin@internal-authz) >> 2018-09-12 14:14:31,006+03 INFO >> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] >> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Running >> command: TestProviderConnectivityCommand internal: false. Entities >> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: >> SystemAction group CREATE_STORAGE_POOL with role type ADMIN >> 2018-09-12 14:14:31,058+03 ERROR >> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] >> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Command >> 'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'default' is using 0 threads out of 1, 5 threads waiting for >> tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engine' is using 0 threads out of 500, 16 threads waiting for >> tasks and 0 tasks in queue. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engineScheduled' is using 0 threads out of 100, 100 threads >> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads >> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'hostUpdatesChecker' is using 0 threads out of 5, 2 threads >> waiting for tasks. 2018-09-12 14:15:23,843+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock >> Acquired to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:15:23,849+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] >> Running command: SyncNetworkProviderCommand internal: true. >> 2018-09-12 14:15:23,900+03 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] >> Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:15:23,901+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock >> freed to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' >> >> >> [root@engine ~]# >> cat /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf # >> This file is automatically generated by engine-setup. Please do not >> edit manually [OVN REMOTE] ovn-remote=ssl:127.0.0.1:6641 >> [SSL] >> https-enabled=true >> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem >> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer >> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass >> [OVIRT] >> ovirt-sso-client-secret=Ms7Gw9qNT6IkXu7oA54tDmxaZDIukABV >> ovirt-host=https://engine.set.local:443 >> ovirt-sso-client-id=ovirt-provider-ovn >> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem >> [PROVIDER] >> provider-host=engine.set.local >> >> >>> 12 сент. 2018 г., в 13:59, Dominik Holler <dholler(a)redhat.com> >>> написал(а): >>> >>> On Wed, 12 Sep 2018 13:04:53 +0300 >>> "Mail SET Inc. Group" <mail(a)set-pro.net> wrote: >>> >>>> Hello Dominik! >>>> I have a same issue with OVN provider and SSL >>>> https://www.mail-archive.com/users@ovirt.org/msg47020.html >>>> <https://www.mail-archive.com/users@ovirt.org/msg47020.html> But >>>> certificate changes not helps to resolve it. Maybe you can help me >>>> with this? >>> >>> Sure. Can you please share the relevant lines of >>> ovirt-provider-ovn.log and engine.log, and the information if you >>> are using the certificates generated by engine-setup with >>> users(a)ovirt.org ? Thanks, >>> Dominik >>> >> > >

5 13

engine-setup failing on 4.3.2 -> 4.3.3 fails during Engine schema refresh fail
by Edward Berger 18 Jul '23

18 Jul '23

I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings. ... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql [ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed Attaching engine-setup logfile.

4 6

oVirt host deploy: disable multipath
by femi adegoke 27 Mar '23

27 Mar '23

https://bugzilla.redhat.com/show_bug.cgi?id=1472356 There are 2 drives in my install that oVirt is claiming as multipath. I have tried blacklisting at /etc/multipath/ but that hasn't worked. Any better solutions? I need these 2 drives to be non-multipathed.

4 19

Out-of-sync networks can only be detached
by Sakhi Hadebe 27 Feb '23

27 Feb '23

Hi, I have a 3-node oVirt cluster. I have configured 2 logical networks: ovirtmgmt and public. Public logical network is attached in only 2 nodes and failing to attach on the 3rd node with the below error Invalid operation, out-of-sync network 'public' can only be detached. Please have been stuck on this for almost the whole day now. How do I fix this error? -- Regards, Sakhi Hadebe

3 5

Network Address Change
by Paul.LKW 28 Dec '22

28 Dec '22

Hi All: I just has a case, I need to change the oVirt host and engine IP address due to data center decommission I checked in the hosted-engine host there are some files I could change ; in ovirt-hosted-engine/hosted-engine.conf ca_subject="O=simple.com, CN=1.2.3.4" gateway=1.2.3.254 and of course I need to change the ovirtmgmt interface IP too, I think just change the above line could do the tick, but where could I change the other host IP in the cluster ? I think I have to be lost all the host as once changed the hosted-engine host IP as it is in diff. sub net. Does there any command line tools could do that or someone has such experience could share? Best Regards, Paul.LKW

4 6

looking for some ISV backup software which integrates the backup API
by Nathanaël Blanchet 08 Jun '22

08 Jun '22

Hello, We are about to change our backup provider, and I find it is a great chance to choose a full supported ovirt backup solution. I currently use this python script vm-backup-scheduler (https://github.com/wefixit-AT/oVirtBackup) but it is not the workflow officially suggested by the community (https://www.ovirt.org/develop/release-management/features/storage/backup-re…) I've been looking for a long time an ISV who supports such an API, but the only one I found is this one : Acronis Backup Advanced suggested here https://access.redhat.com/ecosystem/search/#/ecosystem/Red%20Hat%20Enterpri… I ran the trial version, but it doesn't seem to do better than the vm-backup-scheduler script, and it doesn't seem to use the backup API (attach a clone as a disk to an existing vm). Can you suggest me some other ISV solutions, if they ever exist... or share me your backup experience?

5 4

OVS switch type for hosted-engine
by Devin A. Bougie 06 May '22

06 May '22

Is it possible to setup a hosted engine using the OVS switch type instead of Legacy? If it's not possible to start out as OVS, instructions for switching from Legacy to OVS after the fact would be greatly appreciated. Many thanks, Devin

4 5

USB3 redirection
by Rik Theys 19 Apr '22

19 Apr '22

Hi, I'm trying to assign a USB3 controller to a CentOS 7.4 VM in oVirt 4.1 with USB redirection enabled. I've created the following file in /etc/ovirt-engine/osinfo.conf.d: 01-usb.properties with content os.other.devices.usb.controller.value = nec-xhci and have restarted ovirt-engine. If I disable USB-support in the web interface for the VM, the xhci controller is added to the VM (I can see it in the qemu-kvm commandline), but usb redirection is not available. If I enable USB-support in the UI, no xhci controller is added (only 4 uhci controllers). Is there a way to make the controllers for usb redirection xhci controllers? Regards, Rik -- Rik Theys System Engineer KU Leuven - Dept. Elektrotechniek (ESAT) Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee +32(0)16/32.11.07 ---------------------------------------------------------------- <<Any errors in spelling, tact or fact are transmission errors>>

3 5

OVN routing and firewalling in oVirt
by Gianluca Cecchi 19 Apr '22

19 Apr '22

Hello, how do we manage routing between different OVN networks in oVirt? And between OVN networks and physical ones? Based on architecture read here: http://openvswitch.org/support/dist-docs/ovn-architecture.7.html I see terms for logical routers and gateway routers respectively but how to apply to oVirt configuration? Do I have to choose between setting up a specialized VM or a physical one: is it applicable/advisable to put on oVirt host itself the gateway functionality? Is there any security policy (like security groups in Openstack) to implement? Thanks, Gianluca

4 6

Install hosted-engine - Task Get local VM IP failed
by florentl 10 Mar '22

10 Mar '22

Hi all, I try to install hosted-engine on node : ovirt-node-ng-4.2.3-0.20180518. Every times I get stuck on : [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 50, "changed": true, "cmd": "virsh -r net-dhcp-leases default | grep -i 00:16:3e:6c:5a:91 | awk '{ print $5 }' | cut -f1 -d'/'", "delta": "0:00:00.108872", "end": "2018-06-01 11:17:34.421769", "rc": 0, "start": "2018-06-01 11:17:34.312897", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} I tried with static IP Address and with DHCP but both failed. To be more specific, I installed three nodes, deployed glusterfs with the wizard. I'm in a nested virtualization environment for this lab (Vmware Esxi Hypervisor). My node IP is : 192.168.176.40 / and I want the hosted-engine vm has 192.168.176.43. Thanks, Florent

7 17

Host needs to be reinstalled after configuring power management
by Andrew DeMaria 02 Feb '22

02 Feb '22

Hi, I am running ovirt 4.3 and have found the following action item immediately after configuring power management for a host: Host needs to be reinstalled as important configuration changes were applied on it. The thing is - I've just freshly installed this host and it seems strange that I need to reinstall it. Is there a better way to install a host and configure power management without having to reinstall it after? Thanks, Andrew

4 3

Import an exported VM using Ansible
by paolo＠airaldi.it 25 Jan '22

25 Jan '22

Hello everybody! I'm trying to automate a copy of a VM from one Datacenter to another using an Ansible.playbook. I'm able to: - Create a snapshot of the source VM - create a clone from the snapshot - remove the snapshot - attach an Export Domain - export the clone to the Export Domain - remove the clone - detach the Export domain from the source Datacenter and attach to the destination. Unfortunately I cannot find a module to: - import the VM from the Export Domain - delete the VM image from the Export Domain. Any hint on how to do that? Thanks in advance. Cheers. Paolo PS: if someone is interested I can share the playbook.

4 3

did 4.3.9 reset bug https://bugzilla.redhat.com/show_bug.cgi?id=1590266
by kelley bryan 10 Jan '22

10 Jan '22

I am experiencing the error message in the ovirt-hosted-engine-setup-ansible-create_target_vm log {2020-05-06 14:15:30,024-0500 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u"Fail if Engine IP is different from engine's he_fqdn resolved IP", 'ansible_result': u'type: <type \'dict\'>\nstr: {\'msg\': u"Engine VM IP address is while the engine\'s he_fqdn ovirt1-engine.kelleykars.org resolves to 192.168.122.2. If you are using DHCP, check your DHCP reservation configuration", \'changed\': False, \'_ansible_no_log\': False}', 'task_duration': 1, 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml'}}:Q! The bug 1590266 says it should report the engine VM IP address xxx.xxx.xxx.xxx while the Engines he_fqdn is xxxxxxxxx I need to see what it thins is wrong as both dig fqdn engine name and dig -x ip return the correct information. Now this bug looks like it may play but I don't see the failed rediness check in the this log https://access.redhat.com/solutions/4462431 or is it because the vm fails or dies or ???

3 7

Lots of storage.MailBox.SpmMailMonitor
by Fabrice Bacchella 06 Jan '22

06 Jan '22

My vdsm log files are huge: -rw-r--r-- 1 vdsm kvm 1.8G Nov 22 11:32 vdsm.log And this is juste half an hour of logs: $ head -1 vdsm.log 2018-11-22 11:01:12,132+0100 ERROR (mailbox-spm) [storage.MailBox.SpmMailMonitor] mailbox 2 checksum failed, not clearing mailbox, clearing new mail (data='...lots of data', expected='\xa4\x06\x08\x00') (mailbox:612) I just upgraded vdsm: $ rpm -qi vdsm Name : vdsm Version : 4.20.43

3 5

How to renew an Ovirt host certificate (vdsmcert.pem) ?
by noua.toukourou＠uni.lu 19 Sep '21

19 Sep '21

Hi, I have an Ovirt host that the vdsmcert.pem expired. The problem is that host contains the self-hosted engine. How to renew the certificate without breaking the self-hosted engine ? Thanks,

3 3

Template for Ubuntu 18.04 Server Issues
by jeremy_tourville＠hotmail.com 02 Sep '21

02 Sep '21

I have built a system as a template on oVirt. Specifically, Ubuntu 18.04 server. I am noticing an issue when creating new vms from that template. I used the check box for "seal template" when creating the template. When I create a new Ubuntu VM I am getting duplicate IP addresses for all the machines created from the template. It seems like the checkbox doesn't fully function as intended. I would need to do further manual steps to clear up this issue. Has anyone else noticed this behavior? Is this expected or have I missed something? Thanks for your input!

6 7

Libgfapi considerations
by Jayme 01 Jul '21

01 Jul '21

Are there currently any known issues with using libgfapi in the latest stable version of ovirt in hci deployments? I have recently enabled it and have noticed a significant (over 4x) increase in io performance on my vms. I’m concerned however since it does not seem to be an ovirt default setting. Is libgfapi considered safe and stable to use in ovirt 4.3 hci?

9 16

poweroff and reboot with ovirt_vm ansible module
by Nathanaël Blanchet 18 May '21

18 May '21

Hello, is there a way to poweroff or reboot (without stopped and running state) a vm with the ovirt_vm ansible module? -- Nathanaël Blanchet Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

7 11

supervdsm failing during network_caps
by Alan G 24 Mar '21

24 Mar '21

Hi, I have issues with one host where supervdsm is failing in network_caps. I see the following trace in the log. MainProcess|jsonrpc/1::ERROR::2020-01-06 03:01:05,558::supervdsm_server::100::SuperVdsm.ServerCallback::(wrapper) Error in network_caps Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 98, in wrapper res = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 56, in network_caps return netswitch.configurator.netcaps(compatibility=30600) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 317, in netcaps net_caps = netinfo(compatibility=compatibility) File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 325, in netinfo _netinfo = netinfo_get(vdsmnets, compatibility) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 150, in get return _stringify_mtus(_get(vdsmnets)) File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/cache.py", line 59, in _get ipaddrs = getIpAddrs() File "/usr/lib/python2.7/site-packages/vdsm/network/netinfo/addresses.py", line 72, in getIpAddrs for addr in nl_addr.iter_addrs(): File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/addr.py", line 33, in iter_addrs with _nl_addr_cache(sock) as addr_cache: File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/__init__.py", line 92, in _cache_manager cache = cache_allocator(sock) File "/usr/lib/python2.7/site-packages/vdsm/network/netlink/libnl.py", line 469, in rtnl_addr_alloc_cache raise IOError(-err, nl_geterror(err)) IOError: [Errno 16] Message sequence number mismatch A restart of supervdsm will resolve the issue for a period, maybe 24 hours, then it will occur again. So I'm thinking it's resource exhaustion or a leak of some kind? Running 4.2.8.2 with VDSM at 4.20.46. I've had a look through the bugzilla and can't find an exact match, closest was this one https://bugzilla.redhat.com/show_bug.cgi?id=1666123 which seems to be a RHV only fix. Thanks, Alan

2 3

OVN and change of mgmt network
by Gianluca Cecchi 26 Jan '21

26 Jan '21

Hello, I previously had OVN running on engine (as OVN provider with northd and northbound and southbound DBs) and hosts (with OVN controller). After changing mgmt ip of hosts (engine has retained instead the same ip), I executed again on them the command: vdsm-tool ovn-config <ip_of_engine> <nel_local_ip_of_host> Now I think I have to clean up some things, eg: 1) On engine where I get these lines below systemctl status ovn-northd.service -l . . . Sep 29 14:41:42 ovmgr1 ovsdb-server[940]: ovs|00005|reconnect|ERR|tcp: 10.4.167.40:37272: no response to inactivity probe after 5 seconds, disconnecting Oct 03 11:52:00 ovmgr1 ovsdb-server[940]: ovs|00006|reconnect|ERR|tcp: 10.4.167.41:52078: no response to inactivity probe after 5 seconds, disconnecting The two IPs are the old ones of two hosts It seems that a restart of the services has fixed... Can anyone confirm if I have to do anything else? 2) On hosts (there are 3 hosts with OVN on ip 10.4.192.32/33/34) where I currently have this output [root@ov301 ~]# ovs-vsctl show 3a38c5bb-0abf-493d-a2e6-345af8aedfe3 Bridge br-int fail_mode: secure Port "ovn-1dce5b-0" Interface "ovn-1dce5b-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.32"} Port "ovn-ddecf0-0" Interface "ovn-ddecf0-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.192.33"} Port "ovn-fd413b-0" Interface "ovn-fd413b-0" type: geneve options: {csum="true", key=flow, remote_ip="10.4.168.74"} Port br-int Interface br-int type: internal ovs_version: "2.7.2" [root@ov301 ~]# The IPs of kind 10.4.192.x are ok. But there is a left-over of an old host I initially used for tests, corresponding to 10.4.168.74, that now doesn't exist anymore How can I clean records for 1) and 2)? Thanks, Gianluca

3 4

CentOS Stream support
by Michal Skrivanek 25 Jan '21

25 Jan '21

Hi all, we would like to ask about interest in community about oVirt moving to CentOS Stream. There were some requests before but it’s hard to see how many people would really like to see that. With CentOS releases lagging behind RHEL for months it’s interesting to consider moving to CentOS Stream as it is much more up to date and allows us to fix bugs faster, with less workarounds and overhead for maintaining old code. E.g. our current integration tests do not really pass on CentOS 8.1 and we can’t really do much about that other than wait for more up to date packages. It would also bring us closer to make oVirt run smoothly on RHEL as that is also much closer to Stream than it is to outdated CentOS. So..would you like us to support CentOS Stream? We don’t really have capacity to run 3 different platforms, would you still want oVirt to support CentOS Stream if it means “less support” for regular CentOS? There are some concerns about Stream being a bit less stable, do you share those concerns? Thank you for your comments, michal

5 8

encrypted GENEVE traffic
by Pavel Nakonechnyi 18 Jan '21

18 Jan '21

Dear oVirt Community, From my understanding oVirt does not support Open vSwitch IPSEC tunneling for GENEVE traffic (which is described on pages http://docs.openvswitch.org/en/latest/howto/ipsec/ and http://docs.openvswitch.org/en/latest/tutorials/ipsec/) Are there plans to introduce such support? (or explicitly not to..) Is it possible to somehow manually configure such tunneling for existing virtual networks? (even in a limited way) Alternatively, is it possible to deploy oVirt on top of the tunneled (i.e. via VXLAN, IPSec) interfaces? This will allow to encrypt all management traffic. Such requirement arises when using oVirt deployment on third-party premises with untrusted network. Thank in advance for any clarifications. :) -- WBR, Pavel +32478910884

2 7

oVirt 4.4: Self-hosted engine deployment fails with backup restore from 4.3 engine
by Oliver Leinfelder 11 Dec '20

11 Dec '20

Hi there, I'm a bit puzzled about an possible upgrade paths from a 4.3 cluster to version 4.4 in a self-hosted engine environment. My idea was: Set up a new host with a clean ovirt node 4.4 installation, then deploy the hosted engine on this with a restored backup from the production cluster and go from there. This however fails with the following error: 2020-05-27 00:17:08,886+0200 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:103 {'msg': 'non-zero return code', 'cmd': ['engine-setup', '--accept-defaults', '--config-append=/root/ovirt-engine-answers'], 'stdout': "[ INFO ] Stage: Initializing\n[ INFO ] Stage: Environment setup\n C onfiguration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf, /root/ovirt-engine-answers\n Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200527001657-fyeueu.log\n Version: otop i-1.9.1 (otopi-1.9.1-1.el8)\n[ INFO ] DNF Downloading 1 files, 0.00KB\n[ INFO ] DNF Downloaded CentOS-8 - AppStream\n[ INFO ] DNF Downloading 1 files, 0.00KB\n[ INFO ] DNF Downloaded CentOS-8 - Base\n[ INFO ] DNF Downloading 1 files, 0.00KB\n [...] ... anwsers from backup config follow .... [...] 2020-05-27 00:17:12,396+0200 DEBUG otopi.context context._executeMethod:145 method exception Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 403, in _closeup r = ah.run() File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/ansible_utils.py", line 229, in run raise RuntimeError(_('Failed executing ansible-playbook')) Is this approach (restoring from 4.3) generally supposed to work? If not, what is the appropriate upgrade path? Thank you! Regards Oli

3 7

Can't import some VMs after storage domain detach and reattach to new datacenter.
by m black 10 Dec '20

10 Dec '20

7 14

"gluster-ansible-roles is not installed on Host" error on Cockpit
by Hesham Ahmed 26 Nov '20

26 Nov '20

On a new 4.3.1 oVirt Node installation, when trying to deploy HCI (also when trying adding a new gluster volume to existing clusters) using Cockpit, an error is displayed "gluster-ansible-roles is not installed on Host. To continue deployment, please install gluster-ansible-roles on Host and try again". There is no package named gluster-ansible-roles in the repositories: [root@localhost ~]# yum install gluster-ansible-roles Loaded plugins: enabled_repos_upload, fastestmirror, imgbased-persist, package_upload, product-id, search-disabled-repos, subscription-manager, vdsmupgrade This system is not registered with an entitlement server. You can use subscription-manager to register. Loading mirror speeds from cached hostfile * ovirt-4.3-epel: mirror.horizon.vn No package gluster-ansible-roles available. Error: Nothing to do Uploading Enabled Repositories Report Cannot upload enabled repos report, is this client registered? This is due to check introduced here: https://gerrit.ovirt.org/#/c/98023/1/dashboard/src/helpers/AnsibleUtil.js Changing the line from: [ "rpm", "-qa", "gluster-ansible-roles" ], { "superuser":"require" } to [ "rpm", "-qa", "gluster-ansible" ], { "superuser":"require" } resolves the issue. The above code snippet is installed at /usr/share/cockpit/ovirt-dashboard/app.js on oVirt node and can be patched by running "sed -i 's/gluster-ansible-roles/gluster-ansible/g' /usr/share/cockpit/ovirt-dashboard/app.js && systemctl restart cockpit"

2 1

Error exporting into ova
by Gianluca Cecchi 31 Aug '20

31 Aug '20

Hello, I'm playing with export_vm_as_ova.py downloaded from the examples github: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/export_v… My environment is oVirt 4.3.3.7 with iSCSI storage domain. It fails leaving an ova.tmp file In webadmin gui: Starting to export Vm enginecopy1 as a Virtual Appliance 7/19/1911:55:12 AM VDSM ov301 command TeardownImageVDS failed: Cannot deactivate Logical Volume: ('General Storage Exception: ("5 [] [\' Logical volume fa33df49-b09d-4f86-9719-ede649542c21/0420ef47-0ad0-4cf9-babd-d89383f7536b in use.\']\\nfa33df49-b09d-4f86-9719-ede649542c21/[\'a7480dc5-b5ca-4cb3-986d-77bc12165be4\', \'0420ef47-0ad0-4cf9-babd-d89383f7536b\']",)',) 7/19/1912:25:36 PM Failed to export Vm enginecopy1 as a Virtual Appliance to path /save_ova/base/dump/myvm2.ova on Host ov301 7/19/1912:25:37 PM During export I have this qemu-img process creating the disk over the loop device: root 30878 30871 0 11:55 pts/2 00:00:00 su -p -c qemu-img convert -T none -O qcow2 '/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b' '/dev/loop1' vdsm vdsm 30882 30878 10 11:55 ? 00:00:00 qemu-img convert -T none -O qcow2 /rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b /dev/loop1 The ova.tmp file is getting filled while command runs eg: [root@ov301 ]# du -sh /save_ova/base/dump/myvm2.ova.tmp 416M /save_ova/base/dump/myvm2.ova.tmp [root@ov301 sysctl.d]# [root@ov301 sysctl.d]# du -sh /save_ova/base/dump/myvm2.ova.tmp 911M /save_ova/base/dump/myvm2.ova.tmp [root@ov301 ]# and the final generated / not completed file is in this state: [root@ov301 ]# qemu-img info /save_ova/base/dump/myvm2.ova.tmp image: /save_ova/base/dump/myvm2.ova.tmp file format: raw virtual size: 30G (32217446400 bytes) disk size: 30G [root@ov301 sysctl.d]# But I notice that the timestamp of the file is about 67 minutes after start of job and well after the notice of its failure.... [root@ov301 sysctl.d]# ll /save_ova/base/dump/ total 30963632 -rw-------. 1 root root 32217446400 Jul 19 13:02 myvm2.ova.tmp [root@ov301 sysctl.d]# [root@ov301 sysctl.d]# du -sh /save_ova/base/dump/myvm2.ova.tmp 30G /save_ova/base/dump/myvm2.ova.tmp [root@ov301 sysctl.d]# In engine.log the first error I see is 30 minutes after start 2019-07-19 12:25:31,563+02 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible playbook execution failed: Timeout occurred while executing Ansible playbook. 2019-07-19 12:25:31,563+02 INFO [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Ansible playbook command has exited with value: 1 2019-07-19 12:25:31,564+02 ERROR [org.ovirt.engine.core.bll.CreateOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] Failed to create OVA. Please check logs for more details: /var/log/ovirt-engine/ova/ovirt-export-ova-ansible-20190719115531-ov301-2001ddf4.log 2019-07-19 12:25:31,565+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.TeardownImageVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] START, TeardownImageVDSCommand(HostName = ov301, ImageActionsVDSCommandParameters:{hostId='8ef1ce6f-4e38-486c-b3a4-58235f1f1d06'}), log id: 3d2246f7 2019-07-19 12:25:36,569+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [2001ddf4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ov301 command TeardownImageVDS failed: Cannot deactivate Logical Volume: ('General Storage Exception: ("5 [] [\' Logical volume fa33df49-b09d-4f86-9719-ede649542c21/0420ef47-0ad0-4cf9-babd-d89383f7536b in use.\']\\nfa33df49-b09d-4f86-9719-ede649542c21/[\'a7480dc5-b5ca-4cb3-986d-77bc12165be4\', \'0420ef47-0ad0-4cf9-babd-d89383f7536b\']",)',) In ansible playbook suggested log file I don't see anything useful. It ends with timestamps when the script has been launched. Last lines are: 2019-07-19 11:55:33,877 p=5699 u=ovirt | TASK [ovirt-ova-export-pre-pack : Retrieving the temporary path for the OVA file] *** 2019-07-19 11:55:34,198 p=5699 u=ovirt | changed: [ov301] => { "changed": true, "dest": "/save_ova/base/dump/myvm2.ova.tmp", "gid": 0, "group": "root", "mode": "0600", "owner": "root", "secontext": "system_u:object_r:nfs_t:s0", "size": 32217446912, "state": "file", "uid": 0 } 2019-07-19 11:55:34,204 p=5699 u=ovirt | TASK [ovirt-ova-pack : Run packing script] ************************************* It seems 30 minutes... for timeout? About what, ansible job? Or possibly implicit user session created when running the python script? The snapshot has been correctly deleted (as I see also in engine.log), I don't see it in webadmin gui. Any known problem? Just for test I executed again at 14:24 and I see same Ansible error at 14:54 The snapshot gets deleted, while the qemu-img command still continues.... [root@ov301 sysctl.d]# ps -ef | grep qemu-img root 13504 13501 0 14:24 pts/1 00:00:00 su -p -c qemu-img convert -T none -O qcow2 '/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b' '/dev/loop0' vdsm vdsm 13505 13504 3 14:24 ? 00:01:26 qemu-img convert -T none -O qcow2 /rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/59a4a324-4c99-4ff5-abb1-e9bbac83292a/0420ef47-0ad0-4cf9-babd-d89383f7536b /dev/loop0 root 17587 24530 0 15:05 pts/0 00:00:00 grep --color=auto qemu-img [root@ov301 sysctl.d]# [root@ov301 sysctl.d]# du -sh /save_ova/base/dump/myvm2.ova.tmp 24G /save_ova/base/dump/myvm2.ova.tmp [root@ov301 sysctl.d]# ll /save_ova/base/dump/myvm2.ova.tmp -rw-------. 1 root root 32217446400 Jul 19 15:14 /save_ova/base/dump/myvm2.ova.tmp [root@ov301 sysctl.d]# and then continues until image copy completes, but at this time the job has already aborted and so the completion of the ova composition doesn't go ahead... and I remain with the ova.tmp file... How to extend timeout? Thanks in advance, Gianluca

4 11

deprecating export domain?
by Charles Kozler 30 Aug '20

30 Aug '20

Hello, I recently read on this list from a redhat member that export domain is either being deprecated or looking at being deprecated To that end, can you share details? Can you share any notes/postings/bz's that document this? I would imagine something like this would be discussed in larger audience This seems like a somewhat significant change to make and I am curious where this is scheduled? Currently, a lot of my backups rely explicitly on an export domain for online snapshots, so I'd like to plan accordingly Thanks!

11 21

Upgrade Memory of oVirt Nodes
by souvaliotimaria＠mail.com 19 Aug '20

19 Aug '20

Hello everyone, I have an oVirt 4.3.2.5 hyperconverged 3 node production environment and we want to add some RAM to it. Can I upgrade the RAM without my users noticing any disruptions and keep the VMs running? The way I thought I should do it was to migrate any running VMs to the other nodes, then set one node in maintenance mode, shut it down, place the new memory, bring it back up, remove it from maintenance mode and see how the installation reacts and repeat for the other two nodes. Is this correct or should I follow another way? Will there be a problem during the time when the nodes will not be identical in their resources? Thank you for your time, Souvalioti Maria

4 5

Support for Shared SAS storage
by Vinícius Ferrão 13 Aug '20

13 Aug '20

Hello, I’ve two compute nodes with SAS Direct Attached sharing the same disks. Looking at the supported types I can’t see this on the documentation: https://www.ovirt.org/documentation/admin-guide/chap-Storage.html There’s is local storage on this documentation, but my case is two machines, both using SAS, connected to the same machines. It’s the VRTX hardware from Dell. Is there any support for this? It should be just like Fibre Channel and iSCSI, but with SAS instead. Thanks,

7 11

oVirt 4.4.0 Release is now generally available
by Sandro Bonazzola 12 Aug '20

12 Aug '20

oVirt 4.4.0 Release is now generally available The oVirt Project is excited to announce the general availability of the oVirt 4.4.0 Release, as of May 20th, 2020 This release unleashes an altogether more powerful and flexible open source virtualization solution that encompasses hundreds of individual changes and a wide range of enhancements across the engine, storage, network, user interface, and analytics, as compared to oVirt 4.3. Important notes before you install / upgrade Some of the features included in the oVirt 4.4.0 release require content that will be available in CentOS Linux 8.2 but cannot be tested on RHEL 8.2 yet due to some incompatibility in the openvswitch package that is shipped in CentOS Virt SIG, which requires rebuilding openvswitch on top of CentOS 8.2. The cluster switch type OVS is not implemented for CentOS 8 hosts. Please note that oVirt 4.4 only supports clusters and datacenters with compatibility version 4.2 and above. If clusters or datacenters are running with an older compatibility version, you need to upgrade them to at least 4.2 (4.3 is recommended). Please note that in RHEL 8 / CentOS 8 several devices that worked on EL7 are no longer supported. For example, megaraid_sas driver is removed. If you use Enterprise Linux 8 hosts you can try to provide the necessary drivers for the deprecated hardware using the DUD method (See users mailing list thread on this at https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NDSVUZSESOXEFJ… ) Installation instructions For the engine: either use the oVirt appliance or install CentOS Linux 8 minimal by following these steps: - Install the CentOS Linux 8 image from http://centos.mirror.garr.it/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x8… - dnf install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm - dnf update (reboot if needed) - dnf module enable -y javapackages-tools pki-deps postgresql:12 - dnf install ovirt-engine - engine-setup For the nodes: Either use oVirt Node ISO or: - Install CentOS Linux 8 from http://centos.mirror.garr.it/centos/8.1.1911/isos/x86_64/CentOS-8.1.1911-x8…, selecting the minimal installation. - dnf install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm - dnf update (reboot if needed) - Attach the host to the engine and let it be deployed. Update instructionsUpdate from oVirt 4.4 Release Candidate On the engine side and on CentOS hosts, you’ll need to switch from ovirt44-pre to ovirt44 repositories. In order to do so, you need to: 1. dnf remove ovirt-release44-pre 2. rm -f /etc/yum.repos.d/ovirt-4.4-pre-dependencies.repo 3. rm -f /etc/yum.repos.d/ovirt-4.4-pre.repo 4. dnf install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm 5. dnf update On the engine side you’ll need to run engine-setup only if you were not already on the latest release candidate. On oVirt Node, you’ll need to upgrade with: 1. Move node to maintenance 2. dnf install https://resources.ovirt.org/pub/ovirt-4.4/rpm/el8/noarch/ovirt-node-ng-imag… 3. Reboot 4. Activate the host Update from oVirt 4.3 oVirt 4.4 is available only for CentOS 8. In-place upgrades from previous installations, based on CentOS 7, are not possible. For the engine, use backup, and restore that into a new engine. Nodes will need to be reinstalled. A 4.4 engine can still manage existing 4.3 hosts, but you can’t add new ones. For a standalone engine, please refer to upgrade procedure at https://ovirt.org/documentation/upgrade_guide/#Upgrading_from_4-3 If needed, run ovirt-engine-rename (see engine rename tool documentation at https://www.ovirt.org/documentation/admin-guide/chap-Utilities.html ) When upgrading hosts: You need to upgrade one host at a time. 1. Turn host to maintenance. Virtual machines on that host should migrate automatically to a different host. 2. Remove it from the engine 3. Re-install it with el8 or oVirt Node as per installation instructions 4. Re-add the host to the engine Please note that you may see some issues live migrating VMs from el7 to el8. If you hit such a case, please turn off the vm on el7 host and get it started on the new el8 host in order to be able to move the next el7 host to maintenance. What’s new in oVirt 4.4.0 Release? - Hypervisors based on CentOS Linux 8 (rebuilt from award winning RHEL8), for both oVirt Node and standalone CentOS Linux hosts. - Easier network management and configuration flexibility with NetworkManager. - VMs based on a more modern Q35 chipset with legacy SeaBIOS and UEFI firmware. - Support for direct passthrough of local host disks to VMs. - Live migration improvements for High Performance guests. - New Windows guest tools installer based on WiX framework now moved to VirtioWin project. - Dropped support for cluster level prior to 4.2. - Dropped API/SDK v3 support deprecated in past versions. - 4K block disk support only for file-based storage. iSCSI/FC storage do not support 4K disks yet. - You can export a VM to a data domain. - You can edit floating disks. - Ansible Runner (ansible-runner) is integrated within the engine, enabling more detailed monitoring of playbooks executed from the engine. - Adding and reinstalling hosts is now completely based on Ansible, replacing ovirt-host-deploy, which is not used anymore. - The OpenStack Neutron Agent cannot be configured by oVirt anymore, it should be configured by TripleO instead. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 8.1 * CentOS Linux (or similar) 8.1 This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 8.1 * CentOS Linux (or similar) 8.1 * oVirt Node 4.4 based on CentOS Linux 8.1 (available for x86_64 only) See the release notes [1] for installation instructions and a list of new features and bugs fixed. If you manage more than one oVirt instance, OKD or RDO we also recommend to try ManageIQ <http://manageiq.org/>. In such a case, please be sure to take the qc2 image and not the ova image. Notes: - oVirt Appliance is already available for CentOS Linux 8 - oVirt Node NG is already available for CentOS Linux 8 Additional Resources: * Read more about the oVirt 4.4.0 release highlights: http://www.ovirt.org/release/4.4.0/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.4.0/ [2] http://resources.ovirt.org/pub/ovirt-4.4/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> [image: |Our code is open_] <https://www.redhat.com/en/our-code-is-open> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

11 35

4.4 HCI Install Failure - Missing /etc/pki/CA/cacert.pem
by Stephen Panicho 09 Aug '20

09 Aug '20

Hi all! I'm using Cockpit to perform an HCI install, and it fails at the hosted engine deploy. Libvirtd can't restart because of a missing /etc/pki/CA/cacert.pem file. The log (tasks seemingly from /usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/initial_clean.yml): [ INFO ] TASK [ovirt.hosted_engine_setup : Stop libvirt service] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Drop vdsm config statements] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Restore initial abrt config files] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Restart abrtd service] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Drop libvirt sasl2 configuration by vdsm] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Stop and disable services] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Restore initial libvirt default network configuration] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Start libvirt] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Unable to start service libvirtd: Job for libvirtd.service failed because the control process exited with error code.\nSee \"systemctl status libvirtd.service\" and \"journalctl -xe\" for details.\n"} journalctl -u libvirtd: May 22 04:33:25 node1 libvirtd[26392]: libvirt version: 5.6.0, package: 10.el8 (CBS <cbs(a)centos.org>, 2020-02-27-01:09:46, ) May 22 04:33:25 node1 libvirtd[26392]: hostname: node1 May 22 04:33:25 node1 libvirtd[26392]: Cannot read CA certificate '/etc/pki/CA/cacert.pem': No such file or directory May 22 04:33:25 node1 systemd[1]: libvirtd.service: Main process exited, code=exited, status=6/NOTCONFIGURED May 22 04:33:25 node1 systemd[1]: libvirtd.service: Failed with result 'exit-code'. May 22 04:33:25 node1 systemd[1]: Failed to start Virtualization daemon. From a fresh CentOS 8.1 minimal install, I've installed the following: - The 4.4 repo - cockpit - ovirt-cockpit-dashboard - vdsm-gluster (providing glusterfs-server and allowing the Gluster Wizard to complete) - gluster-ansible-roles (only on the bootstrap host) I'm not exactly sure what that initial bit of the playbook does. Comparing the bootstrap node with another that has yet to be touched, both /etc/libvirt/libvirtd.conf and /etc/sysconfig/libvirtd are the same on both hosts. Yet the bootstrap host can no longer start libvirtd while the other host can. Neither host has the /etc/pki/CA/cacert.pem file. Please let me know if I can provide any more information. Thanks!

9 15

Deploy host engine error: The task includes an option with an undefined variable
by Angel R. Gonzalez 07 Aug '20

07 Aug '20

Hi all! I'm deploying a host engine in a host node with a 8x Intel(R) Xeon(R) CPU E5410 @ 2.33GHz. The deploy proccess show the next message > [INFO]TASK [ovirt.hosted_engine_setup : Convert CPU model name] > [ERROR]fatal: [localhost]: FAILED! => {"msg": "The task includes an > option with an undefined variable. The error was: 'dict object' has no > attribute ''\n\nThe error appears to be in > '/usr/share/ansible/roles/ovirt.hosted_engine_setup/tasks/create_target_vm/01_create_target_hosted_engine_vm.yml': > line 105, column 15, but may\nbe elsewhere in the file depending on > the exact syntax problem.\n\nThe offending line appears to be:\n\n - > debug: var=server_cpu_dict\n ^ here\n\nThere appears to be both 'k=v' > shorthand syntax and YAML in this task. Only one syntax may be used.\n"} The ansible deploy script in his 105 line show: > - name: Parse server CPU list > set_fact: > server_cpu_dict: "{{ server_cpu_dict | > combine({item.split(':')[1]: item.split(':')[3]}) }}" > with_items: >- > {{ > server_cpu_list.json['values']['system_option_value'][0]['value'].split('; > ')|list|difference(['']) }} > - debug: var=server_cpu_dict I don´t know ansible and i don't how to resolve this issue. Any idea? Thanks in advance, Ángel González.

4 4

oVirt thrashes Docker network during installation
by thomas＠hoberg.net 30 Jul '20

30 Jul '20

I want to run containers and VMs side by side and not necessarily nested. The main reason for that is GPUs, Voltas mostly, used for CUDA machine learning not for VDI, which is what most of the VM orchestrators like oVirt or vSphere seem to focus on. And CUDA drivers are notorious for refusing to work under KVM unless you pay $esla. oVirt is more of a side show in my environment, used to run some smaller functional VMs alongside bigger containers, but also in order to consolidate and re-distribute the local compute node storage as a Gluster storage pool: Kibbutz storage and compute, if you want, very much how I understand the HCI philosophy behind oVirt. The full integration of containers and VMs is still very much on the roadmap I believe, but I was surprised to see that even co-existence seems to be a problem currently. So I set-up a 3-node HCI on CentOS7 (GPU-less and older) hosts and then added additional (beefier GPGPU) CentOS7 hosts, that have been running CUDA workloads on the latest Docker-CE v19 something. The installation works fine, I can migrate VMs to these extra hosts etc., but to my dismay Docker containers on these hosts lose access to the local network, that is the entire subnet the host is in. For some strange reason I can still ping Internet hosts, perhaps even everything behind the host's gateway, but local connections are blocked. It would seem that the ovritmgmt network that the oVirt installation puts in breaks the docker0 bridge that Docker put there first. I'd consider that a bug, but I'd like to gather some feedback first, if anyone else has run into this problem. I've repeated this several times in completely distinct environments with the same results: Simply add a host with a working Docker-CE as an oVirt host to an existing DC/cluster and then try if you can still ping anyone on that net, including the Docker host from a busybox container afterwards (should try that ping just before you actually add it). No, I didn't try this with podman yet, because that's separate challenge with CUDA: Would love to know if that is part of QA for oVirt already.

5 9

Non storage nodes erronously included in quota calculations for HCI?
by thomas＠hoberg.net 30 Jul '20

30 Jul '20

For my home-lab I operate a 3 node HCI cluster on 100% passive Atoms, mostly to run light infrastructure services such as LDAP and NextCloud. I then add workstations or even laptops as pure compute hosts to the cluster for bigger but temporary things, that might actually run a different OS most of the time or just be shut off. From oVirt's point of view, these are just first put into maintenance and then shut down until needed again. No fencing or power management, all manual. All nodes, even the HCI ones, run CentOS7 with more of a workstation configuration, so updates pile up pretty quickly. After I recently upgraded one of these extra compute nodes, I found my three node HCI cluster not just faltering, but indeed very hard to reactivate at all. The faltering is a distinct issue: I have the impression that reboots of oVirt nodes cause broadcast storms on my rather simplistic 10Gibt L2 switch, which a normal CentOS instance (or any other OS) doesn't, but that's for another post. No what struck me, was that the gluster daemons on the three HCI nodes kept complaining about a lack of quorum long after the network was all back to normal, even if all three of them were there, saw each other perfectly on "gluster show status all", ready and without any healing issues pending at all. Glusterd would complain on all three nodes that there was no quota for the bricks and stop them. That went away as soon as I started one additional compute node, a node that was a gluster peer (because an oVirt host added to a HCI cluster always gets put into the Gluster, even if it's not contributing storage) but had no bricks. Immediately the gluster daemon on the three nodes with contributing bricks would report back good quota and launch the volumes (and thus all the rest of oVirt), even if in terms of *storage bricks* nothing had changed. I am afraid that downing the extra compute-only oVirtNode will bring down the HCI: Clearly not the type of redundancy it's designed to deliver. Evidently such compute-only hosts (and gluster members) get included into some quorum deliberations even if they hold not a single brick, neither storage nor arbitration. To me that seems like a bug, if that is indeed what happens: There I need your advice and suggestions. AFAIK HCI is a late addition to oVirt/RHEV as storage and compute were orginally designed to be completely distinct. In fact there are still remnants of documentation which seem to prohibit using a node for both compute and storage... what HCI is all about. And I have seen compute nodes with "matching" storage (parts of a distinct HCI setup, that was taken down but still had all the storage and Gluster elements operable), being happliy absorbed into a HCI cluster with all Gluster storage appearing in the GUI etc., without any manual creation or inclusion of bricks: Fully automatic (and undocumented)! In that case it makes sense to widen the scope of quota calculations when additional nodes are hyperconverged elements with contributing bricks. It also seems the only way to turn a 3 node HCI into 6 or 9 node one. But if you really just want to add compute nodes without bricks, those can't get "quota votes" without storage to play a role in the redundancy. I can easily imagine the missing "if then else" in the code here, but I was actually very surprised to see those failure and success messages coming from glusterd itself, which to my understanding is pretty unrelated to oVirt on top. Not from the management engine (wasn't running anyway), not from VDSM. Re-creating the scenario is very scary even if I have gone through this three times already, trying to just bring my HCI back up. And then there is so verbose logs all over the place that I'd like some advice which ones I should post. But simply speaking: Gluster peers should get no quota voting rights on volumes unless they contribute bricks. That rule seems broken. Those in the know, please let me know if am on a goose chase or if there is a real issue here that deserves a bug report.

2 5

Re: Ovirt SYSTEM user does not allow deletion of VM networks
by Konstantinos B 25 Jul '20

25 Jul '20

I've deleted them from the cluster as well but re-appear. I've removed ovirt-engine through engine-cleanup installed again and they reappear. Removed ovirt-engine deleted all "ovirt*" files and currenty trying to re-install. I've checked the host's ovs-vsctl bridges and only the desired are shown. So i believe it's an issue on the engine itself.

3 3

Shutdown procedure for single host HCI Gluster
by Gianluca Cecchi 25 Jul '20

25 Jul '20

Hello, I'm testing the single node HCI with ovirt-node-ng 4.3.9 iso. Very nice and many improvements over the last time I tried it. Good! I have a doubt related to shutdown procedure of the server. Here below my steps: - Shutdown all VMs (except engine) - Put into maintenance data and vmstore domains - Enable Global HA Maintenance - Shutdown engine - Shutdown hypervisor It seems that the last step doesn't end and I had to brutally power off the hypervisor. Here the screenshot regarding infinite failure in unmounting /gluster_bricks/engine https://drive.google.com/file/d/1ee0HG21XmYVA0t7LYo5hcFx1iLxZdZ-E/view?usp=… What would be the right step to do before the final shutdown of hypervisor? Thanks, Gianluca

3 10

oVirt install questions
by David White 20 Jul '20

20 Jul '20

I'm reading through all of the documentation at https://ovirt.org/documentation/, and am a bit overwhelmed with all of the different options for installing oVirt. My particular use case is that I'm looking for a way to manage VMs on multiple physical servers from 1 interface, and be able to deploy new VMs (or delete VMs) as necessary. Ideally, it would be great if I could move a VM from 1 host to a different host as well, particularly in the event that 1 host becomes degraded (bad HDD, bad processor, etc...) I'm trying to figure out what the difference is between an oVirt Node and the oVirt Engine, and how the engine differs from the Manager. I get the feeling that `Engine` = `Manager`. Same thing. I further think I understand the Engine to be essentially synonymous with a vCenter VM for ESXi hosts. Is this correct? If so, then what's the difference between the `self-hosted` vs the `stand-alone` engines? oVirt Engine requirements look to be a minimum of 4GB RAM and 2CPUs. oVirt Nodes, on the other hand, require only 2GB RAM. Is this a requirement just for the physical host, or is that how much RAM that each oVirt node process requires? In other words, if I have a physical host with 12GB of physical RAM, will I only be able to allocate 10GB of that to guest VMs? How much of that should I dedicated to the oVirt node processes? Can you install the oVirt Engine as a VM onto an existing oVirt Node? And then connect that same node to the Engine, once the Engine is installed? Reading through the documentation, it also sounds like oVirt Engine and oVirt Node require different versions of RHEL or CentOS. I read that the Engine for oVirt 4.4.0 requires RHEL (or CentOS) 8.2, whereas each Node requires 7.x (although I'll plan to just use the oVirt Node ISO). I'm also wondering about storage. I don't really like the idea of using local storage, but a single NFS server would also be a single point of failure, and Gluster would be too expensive to deploy, so at this point, I'm leaning towards using local storage. Any advice or clarity would be greatly appreciated. Thanks, David Sent with ProtonMail Secure Email.

4 13

Lots of problems with deploying the hosted-engine (ovirt 4.4 | CentOS 8.2.2004)
by jonas 13 Jul '20

13 Jul '20

Hi! I have banged my head against deploying the ovirt 4.4 self-hosted engine on Centos 8.2 for last couple of days. First I was astonished that resources.ovirt.org has no IPv6 connectivity, which made my initial plan for a mostly IPv6-only deployment impossible. CentOS was installed from scratch using the ks.cgf Kickstart file below, which also adds the ovirt 4.4 repo and installs cockpit-ovirt-dashboard & ovirt-engine-appliance. When deploying the hosted-engine from cockpit while logged in as a non-root (although privileged) user, the "(3) Prepare VM" step instantly fails with a nondescript error message and without generating any logs. By using the browser dev tools it was determined that this was because the ansible vars file could not be created as the non-root user did not have write permissions in '/var/lib/ovirt-hosted-engine-setup/cockpit/' . Shouldn't cockpit be capable of using sudo when appropriate, or at least give a more descriptive error message? After login into cockpit as root, or when using the command line ovirt-hosted-engine-setup tool, the deployment fails with "Failed to download metadata for repo 'AppStream'". This seems to be because a) the dnsmasq running on the host does not forward dns queries, even though the host itself can resolve dns queries just fine, and b) there also does not seem to be any functioning routing setup to reach anything outside the host. Regarding a) it is strange that dnsmasq is running with a config file '/var/lib/libvirt/dnsmasq/default.conf' containing the 'no-resolv' option. Could the operation of systemd-resolved be interfering with dnsmasq (see ss -tulpen output)? I tried to manually stop systemd-resolved, but got the same behaviour as before. I hope someone could give me a hint how I could get past this problem, as so far my ovirt experience has been a little bit sub-par. :D Also when running ovirt-hosted-engine-cleanup, the extracted engine VMs in /var/tmp/localvm* are not removed, leading to a "disk-memory-leak" with subsequent runs. Best regards Jonas --- ss -tulpen output post deploy-run --- [root@nxtvirt ~]# ss -tulpen | grep ':53 ' udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=1379,fd=18)) uid:193 ino:32910 sk:6 <-> udp UNCONN 0 0 [fd00:1234:5678:900::1]:53 [::]:* users:(("dnsmasq",pid=13525,fd=15)) uid:979 ino:113580 sk:d v6only:1 <-> udp UNCONN 0 0 [fe80::5054:ff:fe94:f314]%virbr0:53 [::]:* users:(("dnsmasq",pid=13525,fd=12)) uid:979 ino:113575 sk:e v6only:1 <-> tcp LISTEN 0 32 [fd00:1234:5678:900::1]:53 [::]:* users:(("dnsmasq",pid=13525,fd=16)) uid:979 ino:113581 sk:20 v6only:1 <-> tcp LISTEN 0 32 [fe80::5054:ff:fe94:f314]%virbr0:53 [::]:* users:(("dnsmasq",pid=13525,fd=13)) uid:979 ino:113576 sk:21 v6only:1 <-> --- running dnsmasq processes on host ('nxtvirt') post deploy-run --- dnsmasq 13525 0.0 0.0 71888 2344 ? S 12:31 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper root 13526 0.0 0.0 71860 436 ? S 12:31 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper --- var/lib/libvirt/dnsmasq/default.conf --- ##WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE ##OVERWRITTEN AND LOST. Changes to this configuration should be made using: ## virsh net-edit default ## or other application using the libvirt API. ## ## dnsmasq conf file created by libvirt strict-order pid-file=/run/libvirt/network/default.pid except-interface=lo bind-dynamic interface=virbr0 dhcp-option=3 no-resolv ra-param=*,0,0 dhcp-range=fd00:1234:5678:900::10,fd00:1234:5678:900::ff,64 dhcp-lease-max=240 dhcp-hostsfile=/var/lib/libvirt/dnsmasq/default.hostsfile addn-hosts=/var/lib/libvirt/dnsmasq/default.addnhosts enable-ra --- cockpit wizard overview before the 'Prepare VM' step --- VM Engine FQDN:engine.*REDACTED* MAC Address:00:16:3e:20:13:b3 Network Configuration:Static VM IP Address:*REDACTED*:1099:babe::3/64 Gateway Address:*REDACTED*:1099::1 DNS Servers:*REDACTED*:1052::11 Root User SSH Access:yes Number of Virtual CPUs:4 Memory Size (MiB):4096 Root User SSH Public Key:(None) Add Lines to /etc/hosts:yes Bridge Name:ovirtmgmt Apply OpenSCAP profile:no Engine SMTP Server Name:localhost SMTP Server Port Number:25 Sender E-Mail Address:root@localhost Recipient E-Mail Addresses:root@localhost --- ks.cgf --- #version=RHEL8 ignoredisk --only-use=vda autopart --type=lvm # Partition clearing information clearpart --drives=vda --all --initlabel # Use graphical install #graphical text # Use CDROM installation media cdrom # Keyboard layouts keyboard --vckeymap=de --xlayouts='de','us' # System language lang en_US.UTF-8 # Network information network --bootproto=static --device=enp1s0 --ip=192.168.199.250 --netmask=255.255.255.0 --gateway=192.168.199.10 --ipv6=*REDACTED*:1090:babe::250/64 --ipv6gateway=*REDACTED*:1090::1 --hostname=nxtvirt.*REDACTED* --nameserver=*REDACTED*:1052::11 --activate network --hostname=nxtvirt.*REDACTED* # Root password rootpw --iscrypted $6$*REDACTED* firewall --enabled --service=cockpit --service=ssh # Run the Setup Agent on first boot firstboot --enable # Do not configure the X Window System skipx # System services services --enabled="chronyd" # System timezone timezone Etc/UTC --isUtc --ntpservers=ntp.*REDACTED*,ntp2.*REDACTED* user --name=nonrootuser --groups=wheel --password=$6$*REDACTED* --iscrypted # KVM Users/Groups group --name=kvm --gid=36 user --name=vdsm --uid=36 --gid=36 %packages @^server-product-environment #@graphical-admin-tools @headless-management kexec-tools cockpit %end %addon com_redhat_kdump --enable --reserve-mb='auto' %end %anaconda pwpolicy root --minlen=6 --minquality=1 --notstrict --nochanges --notempty pwpolicy user --minlen=6 --minquality=1 --notstrict --nochanges --emptyok pwpolicy luks --minlen=6 --minquality=1 --notstrict --nochanges --notempty %end %post --erroronfail --log=/root/ks-post.log #!/bin/sh dnf update -y # NFS storage mkdir -p /opt/ovirt/nfs-storage chown -R 36:36 /opt/ovirt/nfs-storage chmod 0755 /opt/ovirt/nfs-storage echo "/opt/ovirt/nfs-storage localhost" > /etc/exports echo "/opt/ovirt/nfs-storage engine.*REDACTED*" >> /etc/exports dnf install -y nfs-utils systemctl enable nfs-server.service # Install ovirt packages dnf install -y https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm dnf install -y cockpit-ovirt-dashboard ovirt-engine-appliance # Enable cockpit systemctl enable cockpit.socket %end #reboot --eject --kexec reboot --eject --- Host (nxtvirt) ip -a post deploy-run --- 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:ad:79:1b brd ff:ff:ff:ff:ff:ff inet 192.168.199.250/24 brd 192.168.199.255 scope global noprefixroute enp1s0 valid_lft forever preferred_lft forever inet6 *REDACTED*:1099:babe::250/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fead:791b/64 scope link noprefixroute valid_lft forever preferred_lft forever 5: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:94:f3:14 brd ff:ff:ff:ff:ff:ff inet6 fd00:1234:5678:900::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe94:f314/64 scope link valid_lft forever preferred_lft forever 6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000 link/ether 52:54:00:94:f3:14 brd ff:ff:ff:ff:ff:ff 7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr0 state UNKNOWN group default qlen 1000 link/ether fe:16:3e:68:d3:8a brd ff:ff:ff:ff:ff:ff inet6 fe80::fc16:3eff:fe68:d38a/64 scope link valid_lft forever preferred_lft forever --- iptables-save post deploy-run --- # Generated by iptables-save v1.8.4 on Sun Jun 28 13:20:53 2020 *filter :INPUT ACCEPT [4007:8578553] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [3920:7633249] :LIBVIRT_INP - [0:0] :LIBVIRT_OUT - [0:0] :LIBVIRT_FWO - [0:0] :LIBVIRT_FWI - [0:0] :LIBVIRT_FWX - [0:0] -A INPUT -j LIBVIRT_INP -A FORWARD -j LIBVIRT_FWX -A FORWARD -j LIBVIRT_FWI -A FORWARD -j LIBVIRT_FWO -A OUTPUT -j LIBVIRT_OUT -A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 53 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p udp -m udp --dport 67 -j ACCEPT -A LIBVIRT_INP -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 53 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 53 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT -A LIBVIRT_OUT -o virbr0 -p tcp -m tcp --dport 68 -j ACCEPT -A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable -A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A LIBVIRT_FWX -i virbr0 -o virbr0 -j ACCEPT COMMIT # Completed on Sun Jun 28 13:20:53 2020 # Generated by iptables-save v1.8.4 on Sun Jun 28 13:20:53 2020 *security :INPUT ACCEPT [3959:8576054] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [3920:7633249] COMMIT # Completed on Sun Jun 28 13:20:53 2020 # Generated by iptables-save v1.8.4 on Sun Jun 28 13:20:53 2020 *raw :PREROUTING ACCEPT [4299:8608260] :OUTPUT ACCEPT [3920:7633249] COMMIT # Completed on Sun Jun 28 13:20:53 2020 # Generated by iptables-save v1.8.4 on Sun Jun 28 13:20:53 2020 *mangle :PREROUTING ACCEPT [4299:8608260] :INPUT ACCEPT [4007:8578553] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [3920:7633249] :POSTROUTING ACCEPT [3923:7633408] :LIBVIRT_PRT - [0:0] -A POSTROUTING -j LIBVIRT_PRT COMMIT # Completed on Sun Jun 28 13:20:53 2020 # Generated by iptables-save v1.8.4 on Sun Jun 28 13:20:53 2020 *nat :PREROUTING ACCEPT [337:32047] :INPUT ACCEPT [0:0] :POSTROUTING ACCEPT [159:9351] :OUTPUT ACCEPT [159:9351] :LIBVIRT_PRT - [0:0] -A POSTROUTING -j LIBVIRT_PRT COMMIT # Completed on Sun Jun 28 13:20:53 2020

5 6

Weird problem starting VMs in oVirt-4.4
by Joop 10 Jul '20

10 Jul '20

Hi All, Just had a rather new experience in that starting a VM worked but the kernel entered grub2 rescue console due to the fact that something was wrong with its virtio-scsi disk. The message is Booting from Hard Disk .... error: ../../grub-core/kern/dl.c:266:invalid arch-independent ELF maginc. entering rescue mode... Doing a CTRL-ALT-Del through the spice console let the VM boot correctly. Shutting it down and repeating the procedure I get a disk problem everytime. Weird thing is if I activate the BootMenu and then straight away start the VM all is OK. I don't see any ERROR messages in either vdsm.log, engine.log If I would have to guess it looks like the disk image isn't connected yet when the VM boots but thats weird isn't it? Regards, Joop

10 22

Ovirt 4.3.10 Glusterfs SSD slow performance over 10GE
by jury cat 07 Jul '20

07 Jul '20

Hello all, I am using Ovirt 4.3.10 on Centos 7.8 with glusterfs 6.9 . My Gluster setup is of 3 hosts in replica 3 (2 hosts + 1 arbiter). All the 3 hosts are Dell R720 with Perc Raid Controller H710 mini(that has maximim throughtout 6Gbs) and with 2×1TB samsumg SSD in RAID 0. The volume is partitioned using LVM thin provision and formated XFS. The hosts have separate 10GE network cards for storage traffic. The Gluster Network is connected to this 10GE network cards and is mounted using Fuse Glusterfs(NFS is disabled).Also Migration Network is activated on the same storage network. The problem is that the 10GE network is not used at full potential by the Gluster. If i do live Migration of Vms i can see speeds of 7GB/s ~ 9GB/s. The same network tests using iperf3 reported 9.9GB/s , these exluding the network setup as a bottleneck(i will not paste all the iperf3 tests here for now). I did not enable all the Volume options from "Optimize for Virt Store", because of the bug that cant set volume cluster.granural-heal to enable(this was fixed in vdsm-4 40, but that is working only on Centos 8 with ovirt 4.4 ) . i whould be happy to know what are all these "Optimize for Virt Store" options, so i can set them manually. The speed on the disk inside the host using dd is b etween 1GB/s to 700Mbs. [root@host1 ~]# dd if=/dev/zero of=test bs=100M count=40 cou nt=80 status=progress 8074035200 bytes (8.1 GB) copied, 11.059372 s, 730 MB/s 80+0 records in 80+0 records out 8388608000 bytes (8.4 GB) copied, 11.9928 s, 699 MB/s The dd write test on the gluster volme inside the host is poor only ~ 120MB/s . During the dd test, if i look at Networks->Gluster network ->Hosts at Tx and Rx the network speed barerly reaches over 1Gbs (~1073 Mbs) out of maximum of 10000 Mbs. dd if=/dev/zero of=/rhev/data-center/mnt/glu sterSD/gluster1.domain.local\:_data/test bs=100M count=80 status=progress 8283750400 bytes (8.3 GB) copied, 71.297942 s, 116 MB/s 80+0 records in 80+0 records out 8388608000 bytes (8.4 GB) copied, 71.9545 s, 117 MB/s I have attached my Gluster volume settings and mount options. Thanks, Emy

4 10

oVirt-node 4.4.0 - Hosted engine deployment fails when host is unable to download updates
by Marco Fais 04 Jul '20

04 Jul '20

Hi, fresh installation of oVirt-node 4.4.0 on a cluster -- the hosted-engine --deploy command fails if DNF is unable to download updates. This cluster is not connected to the public network at the moment. If I use a proxy (setting the relevant env. variables) it fails at a later stage (I think the engine VM is trying to download updates as well, but encounters the same issue and doesn't seem to use the proxy). With oVirt-node 4.3.x I didn't have this issue -- any suggestions? [~]# hosted-engine --deploy [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup During customization use CTRL-D to abort. Continuing will configure this host for serving as hypervisor and will create a local VM with a running engine. The locally running engine will be used to configure a new storage domain and create a VM there. At the end the disk of the local VM will be moved to the shared storage. Are you sure you want to continue? (Yes, No)[Yes]: Configuration files: Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20200604214638-vae2wf.log Version: otopi-1.9.1 (otopi-1.9.1-1.el8) [ INFO ] DNF Downloading 1 files, 0.00KB [ INFO ] DNF Downloaded Extra Packages for Enterprise Linux 8 - x86_64 [ * ERROR ] DNF Failed to download metadata for repo 'ovirt-4.4-epel'[ ERROR ] DNF Failed to download metadata for repo 'ovirt-4.4-epel'[ ERROR ] Failed to execute stage 'Environment setup': Failed to download metadata for repo 'ovirt-4.4-epel'* [ INFO ] Stage: Clean up [...] Thanks, Marco

2 2

New fenceType in oVirt code for IBM OpenBMC
by Vinícius Ferrão 04 Jul '20

04 Jul '20

Hello, After some days scratching my head I found that oVirt is probably missing fenceTypes for IBM’s implementation of OpenBMC in the Power Management section. The host machine is an OpenPOWER AC922 (ppc64le). The BMC basically is an “ipmilan” device but the ciphers must be defined as 3 or 17 by default: [root@h01 ~]# ipmitool -I lanplus -H 10.20.10.2 root -P 0penBmc -L operator -C 3 channel getciphers ipmi ID IANA Auth Alg Integrity Alg Confidentiality Alg 3 N/A hmac_sha1 hmac_sha1_96 aes_cbc_128 17 N/A hmac_sha256 sha256_128 aes_cbc_128 The default ipmilan connector forces the option cipher=1 which breaks the communication. So I was reading the code and found this “fenceType” class, but I wasn't able to found where to define those classes. So I can create another one called something like openbmc to set cipher=17 by default. Another question is how bad the output is, it only returns a JSON-RPC generic error. But I don’t know how to suggest a fix for this. Thanks,

2 3

VMs shutdown mysteriously
by Bobby 01 Jul '20

01 Jul '20

Hello, All 4 VMs on one of my oVirt cluster node shutdown for an unknown reason almost simultaneously. Please help me to find the root cause. Thanks. Please note the host seems doing fine and never crash or hangs and I can migrate VMs back to it later. Here is the exact timeline of all the related events combined from the host and the VM(s): On oVirt host: /var/log/vdsm/vdsm.log: 2020-06-25 15:25:16,944-0500 WARN (qgapoller/3) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7f4ed2f9f5f0> on ['e0257b06-28fd-4d41-83a9-adf1904d3622'] (periodic:289) 2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.ovirt-guest-agent.0 already removed (fileutils:54) 2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File: /var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.org.qemu.guest_agent.0 already removed (fileutils:54) [root@athos log]# journalctl -u NetworkManager --since=today -- Logs begin at Wed 2020-05-20 22:07:33 CDT, end at Thu 2020-06-25 16:36:05 CDT. -- Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146] device (vnet0): released from master device SRV-VL /var/log/messages: Jun 25 15:25:18 athos kernel: SRV-VL: port 2(vnet0) entered disabled state Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146] device (vnet0): released from master device SRV-VL Jun 25 15:25:18 athos libvirtd: 2020-06-25 20:25:18.122+0000: 2713: error : qemuMonitorIO:718 : internal error: End of file from qemu monitor /var/log/libvirt/qemu/aries.log: 2020-06-25T20:25:28.353975Z qemu-kvm: terminating on signal 15 from pid 2713 (/usr/sbin/libvirtd) 2020-06-25 20:25:28.584+0000: shutting down, reason=shutdown ============================================================================================= On the first VM effected (same thing on others): /var/log/ovirt-guest-agent/ovirt-guest-agent.log: MainThread::INFO::2020-06-25 15:25:20,270::ovirt-guest-agent::104::root::Stopping oVirt guest agent CredServer::INFO::2020-06-25 15:25:20,626::CredServer::262::root::CredServer has stopped. MainThread::INFO::2020-06-25 15:25:21,150::ovirt-guest-agent::78::root::oVirt guest agent is down. ============================================================================================= Packages version installated: Host OS version: CentOS 7.7.1908: ovirt-hosted-engine-ha-2.3.5-1.el7.noarch ovirt-provider-ovn-driver-1.2.22-1.el7.noarch ovirt-release43-4.3.6-1.el7.noarch ovirt-imageio-daemon-1.5.2-0.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch ovirt-imageio-common-1.5.2-0.el7.x86_64 ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch ovirt-vmconsole-host-1.0.7-2.el7.noarch ovirt-host-4.3.4-1.el7.x86_64 libvirt-4.5.0-23.el7_7.1.x86_64 libvirt-daemon-4.5.0-23.el7_7.1.x86_6 qemu-kvm-ev-2.12.0-33.1.el7.x86_64 qemu-kvm-common-ev-2.12.0-33.1.el7.x86_64 On guest VM: ovirt-guest-agent-1.0.13-1.el6.noarch qemu-guest-agent-0.12.1.2-2.491.el6_8.3.x86_64

4 4

status of oVirt 4.4.x and CentOS 8.2
by Gianluca Cecchi 01 Jul '20

01 Jul '20

Hello, what is the current status both if using plain CentOS based nodes and ovirt-node-ng? Do the release of CentOS 8.2 impact new installation for 4.4.0 and/or 4.4.1rc? Thanks, Gianluca

5 8

Localdisk hook not working
by tim-nospam＠bordemann.com 30 Jun '20

30 Jun '20

Hi, I'd like to use the localdisk hook of vdsm and have configured everything according to the readme: https://github.com/oVirt/vdsm/tree/master/vdsm_hooks/localdisk After installing the hook, configuring the ovirt-engine, creating the volume group, adding the custom property 'localdisk' to the virtual machine and fixing a small bug in the localdisk-helper, vdsm creates the logical volume on the 'ovirt-local' volume group after starting the virtual machine. Unfortunately nothing more seems to happen then. There is no activity on the NAS which would indicate that the disk image is pulled to the host. I can also see no errors on the host. Is anyone currently running ovirt 4.4 with the localdisk hook? What else can I do to find out why the image is not being copied to my host? Thanks, Tim

2 2

Re: VDSM not binding on IPv4 after ovirt-engine restart
by Dominik Holler 30 Jun '20

30 Jun '20

On Tue, Jun 30, 2020 at 6:13 PM Erez Zarum <erezz(a)nanosek.com> wrote: > While troubleshooting a fresh installation of (after a failed one) that > caused all the hosts but the one running the hosted-engine to become in > “Unassigned” state I noticed that the ovirt-engine complains about not > being able to contact the VDSM. > I noticed that VDSM has stopped listening on IPv4. > > Thanks for sharing the details. > I didn’t disable any IPv6 as it states not to disable it on hosts that are > capable running the hosted-engine and it seems that the reason behind it > is that the hosted-engine talks to the host it runs on through “localhost”, > this also explains why the host which the hosted-engine runs on is “OK”. > > Below is from a host that does not run the hosted-engine: > # ss -atn | grep 543 > LISTEN 0 5 *:54322 *:* > ESTAB 0 0 127.0.0.1:54792 127.0.0.1:54321 > ESTAB 0 0 127.0.0.1:54798 127.0.0.1:54321 > LISTEN 0 5 [::]:54321 [::]:* > ESTAB 0 0 [::ffff:127.0.0.1]:54321 > [::ffff:127.0.0.1]:54798 > ESTAB 0 0 [::ffff:127.0.0.1]:54321 > [::ffff:127.0.0.1]:54792 > ESTAB 0 0 [::1]:54321 [::1]:50238 > ESTAB 0 0 [::1]:50238 [::1]:54321 > > Below is from a host that runs the hosted-engine at the moment: > # ss -atn | grep 543 > LISTEN 0 5 *:54322 *:* > LISTEN 0 5 [::]:54321 [::]:* > ESTAB 0 0 [::1]:51230 [::1]:54321 > ESTAB 0 0 [::1]:54321 [::1]:51242 > ESTAB 0 0 [::ffff:10.46.20.23]:54321 > [::ffff:10.46.20.20]:45706 > ESTAB 0 0 [::ffff:10.46.20.23]:54321 > [::ffff:10.46.20.20]:45746 > ESTAB 0 0 [::1]:51240 [::1]:54321 > ESTAB 0 0 [::1]:54321 [::1]:51230 > ESTAB 0 0 [::1]:51242 [::1]:54321 > ESTAB 0 0 [::1]:54321 [::1]:51240 > > The hosted-engine IP is 10.46.20.20 and the host is 10.46.20.23. > > Why do you think the host does not listen to IPv4 anymore? Can you please share the output of "nc -vz 10.46.20.23 54321" executed on engine VM or another host? > /etc/hosts on all hosts: > 127.0.0.1 localhost localhost.localdomain localhost4 > localhost4.localdomain4 > ::1 localhost localhost.localdomain localhost6 > localhost6.localdomain6 > > Perhaps this is relevant but all hosts are enrolled into IDM (FreeIPA) and > as an outcome they all have a DNS record and a PTR record as well as the > ovirt-engine VM. > > # cat /etc/vdsm/vdsm.conf > [vars] > ssl = true > ssl_ciphers = HIGH:!aNULL > ssl_excludes = OP_NO_TLSv1,OP_NO_TLSv1_1 > > [addresses] > management_port = 54321 > > I have tried adding “management_ip = 0.0.0.0” but then it only binds to > IPv4 and yet, the host still shows as Unassigned, sometimes it switches to > “NonResponsive” and trying to “Reinstall” the host fails, the ovirt-engine > complains it can't contact/reach the VDSM, while using netcat from the > ovirt-engine it works. > > I have KSM and Memory Ballooning enabled on the Cluster as well. > > oVirt 4.3.10 installed on CentOS 7.8.2003 > The self-hosted Engine runs on an external GlusterFS, before reinstalling > everything (fresh start of OS, etc..) I tried iSCSI as well. > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/NCTWZLS2VPIAB… >

2 1

VDSM not binding on IPv4 after ovirt-engine restart
by Erez Zarum 30 Jun '20

30 Jun '20

While troubleshooting a fresh installation of (after a failed one) that caused all the hosts but the one running the hosted-engine to become in “Unassigned” state I noticed that the ovirt-engine complains about not being able to contact the VDSM. I noticed that VDSM has stopped listening on IPv4. I didn’t disable any IPv6 as it states not to disable it on hosts that are capable running the hosted-engine and it seems that the reason behind it is that the hosted-engine talks to the host it runs on through “localhost”, this also explains why the host which the hosted-engine runs on is “OK”. Below is from a host that does not run the hosted-engine: # ss -atn | grep 543 LISTEN 0 5 *:54322 *:* ESTAB 0 0 127.0.0.1:54792 127.0.0.1:54321 ESTAB 0 0 127.0.0.1:54798 127.0.0.1:54321 LISTEN 0 5 [::]:54321 [::]:* ESTAB 0 0 [::ffff:127.0.0.1]:54321 [::ffff:127.0.0.1]:54798 ESTAB 0 0 [::ffff:127.0.0.1]:54321 [::ffff:127.0.0.1]:54792 ESTAB 0 0 [::1]:54321 [::1]:50238 ESTAB 0 0 [::1]:50238 [::1]:54321 Below is from a host that runs the hosted-engine at the moment: # ss -atn | grep 543 LISTEN 0 5 *:54322 *:* LISTEN 0 5 [::]:54321 [::]:* ESTAB 0 0 [::1]:51230 [::1]:54321 ESTAB 0 0 [::1]:54321 [::1]:51242 ESTAB 0 0 [::ffff:10.46.20.23]:54321 [::ffff:10.46.20.20]:45706 ESTAB 0 0 [::ffff:10.46.20.23]:54321 [::ffff:10.46.20.20]:45746 ESTAB 0 0 [::1]:51240 [::1]:54321 ESTAB 0 0 [::1]:54321 [::1]:51230 ESTAB 0 0 [::1]:51242 [::1]:54321 ESTAB 0 0 [::1]:54321 [::1]:51240 The hosted-engine IP is 10.46.20.20 and the host is 10.46.20.23. /etc/hosts on all hosts: 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 Perhaps this is relevant but all hosts are enrolled into IDM (FreeIPA) and as an outcome they all have a DNS record and a PTR record as well as the ovirt-engine VM. # cat /etc/vdsm/vdsm.conf [vars] ssl = true ssl_ciphers = HIGH:!aNULL ssl_excludes = OP_NO_TLSv1,OP_NO_TLSv1_1 [addresses] management_port = 54321 I have tried adding “management_ip = 0.0.0.0” but then it only binds to IPv4 and yet, the host still shows as Unassigned, sometimes it switches to “NonResponsive” and trying to “Reinstall” the host fails, the ovirt-engine complains it can't contact/reach the VDSM, while using netcat from the ovirt-engine it works. I have KSM and Memory Ballooning enabled on the Cluster as well. oVirt 4.3.10 installed on CentOS 7.8.2003 The self-hosted Engine runs on an external GlusterFS, before reinstalling everything (fresh start of OS, etc..) I tried iSCSI as well.

1 0