July 2023 - Users - oVirt List Archives

how to renew expired ovirt node vdsm cert manually ?
by dhanaraj.ramesh＠yahoo.com 11 Jan '24

11 Jan '24

below are the steps to renew the expired vdsm cert of ovirt node # To check CERT expired # openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates 1. Backup vdsm folder # cd /etc/pki # mv vdsm vdsm.orig # mkdir vdsm ; chown vdsm:kvm vdsm # cd vdsm # mkdir libvirt-vnc certs keys libvirt-spice libvirt-migrate # chown vdsm:kvm libvirt-vnc certs keys libvirt-spice libvirt-migrate 2. Regenerate cert & keys # vdsm-tool configure --module certificates 3. Copy the cert to destination location chmod 440 /etc/pki/vdsm/keys/vdsmkey.pem chown root /etc/pki/vdsmcerts/*pem chmod 644 /etc/pki/vdsmcerts/*pem cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-spice/server-key.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-vnc/server-key.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem cp -p /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-migrate/ca-cert.pem cp -p /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-migrate/server-key.pem cp -p /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem chown root:qemu /etc/pki/vdsm/libvirt-migrate/server-key.pem cp -p /etc/pki/vdsm.orig/keys/libvirt_password /etc/pki/vdsm/keys/ mv /etc/pki/libvirt/clientcert.pem /etc/pki/libvirt/clientcert.pem.orig mv /etc/pki/libvirt/private/clientkey.pem /etc/pki/libvirt/private/clientkey.pem.orig mv /etc/pki/CA/cacert.pem /etc/pki/CA/cacert.pem.orig cp -p /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/libvirt/clientcert.pem cp -p /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/libvirt/private/clientkey.pem cp -p /etc/pki/vdsm/certs/cacert.pem /etc/pki/CA/cacert.pem 3. cross check the backup folder /etc/pki/vdsm.orig vs /etc/pki/vdsm # refer to /etc/pki/vdsm.orig/*/ and set the correct owner & group permission in /etc/pki/vdsm/*/ 4. restart services # Make sure both services are up systemctl restart vdsmd libvirtd

5 4

Grafana - Origin Not Allowed
by Maton, Brett 14 Dec '23

14 Dec '23

oVirt 4.5.0.8-1.el8 I tried to connect to grafana via the monitoring portal link from the dash and all panels are failing to display any data with varying error messages, but all include 'Origin Not Allowed' I navigated to Data Sources and ran a test on the PostgreSQL connection (localhost) which threw the same Origin Not Allowed error message. Any suggestions?

6 8

Multiple hosts stuck in Connecting state waiting for storage pool to go up.
by ivan.lezhnjov.iv＠gmail.com 06 Nov '23

06 Nov '23

Hi! We have a problem with multiple hosts stuck in Connecting state, which I hoped somebody here could help us wrap our heads around. All hosts, except one, seem to have very similar symptoms but I'll focus on one host that represents the rest. So, the host is stuck in Connecting state and this what we see in oVirt log files. /var/log/ovirt-engine/engine.log: 2023-04-20 09:51:53,021+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-37) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = ABC010-176-XYZ, VdsIdAndVdsVDSCommandParametersBase:{hostId='2c458562-3d4d-4408-afc9-9a9484984a91', vds='Host[ABC010-176-XYZ,2c458562-3d4d-4408-afc9-9a9484984a91]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: SSL session is invalid 2023-04-20 09:55:16,556+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-67) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ABC010-176-XYZ command Get Host Capabilities failed: Message timeout which can be caused by communication issues /var/log/vdsm/vdsm.log: 2023-04-20 17:48:51,977+0300 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList() from=internal, task_id=ebce7c8c-6ded-454e-9aee-86edf72764ef (api:31) 2023-04-20 17:48:51,977+0300 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=ebce7c8c-6ded-454e-9aee-86edf72764ef (api:37) 2023-04-20 17:48:51,978+0300 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:723) Both engine.log and vdsm.log are flooded with these messages. They are repeated at regular intervals ad infinitum. This is one common symptom shared by multiple hosts in our deployment. They all have these message loops in engine.log and vdsm.log files. On all Running vdsm-client Host getConnectedStoragePools also returns an empty list represented by [] on all hosts (but interestingly there is one that showed Storage Pool UUID and yet it was still stuck in Connecting state). This particular host (ABC010-176-XYZ) is connected to 3 CEPH iSCSI Storage Domains and lsblk shows 3 block devices with matching UUIDs in their device components. So, the storage seems to be connected but the Storage Pool is not? How is that even possible? Now, what's even more weird is that we tried rebooting the host (via Administrator Portal) and it didn't help. We even tried removing and re-adding the host in Administrator Portal but to no avail. Additionally, the host refused to go into Maintenance mode so we had to enforce it by manually updating Engine DB. We also tried reinstalling the host via Administrator Portal and ran into another weird problem, which I'm not sure if it's a related one or a problem that deserves a dedicated discussion thread but, basically, the underlying Ansible playbook exited with the following error message: "stdout" : "fatal: [10.10.10.176]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Data could not be sent to remote host \\\"10.10.10.176\\\". Make sure this host can be reached over ssh: \", \"unreachable\": true}", Counterintuitively, just before running Reinstall via Administrator Portal we had been able to reboot the same host (which as you know oVirt does via Ansible as well). So, no changes on the host in between just different Ansible playbooks. To confirm that we actually had access to the host over ssh we successfully ran ssh -p $PORT root(a)10.10.10.176 -i /etc/pki/ovirt-engine/keys/engine_id_rsa and it worked. That made us scratch our heads for a while but what seems to had fixed Ansible's ssh access problems was manual full stop of all VDSM-related systemd services on the host. It was just a wild guess but as soon as we stopped all VDSM services Ansible stopped complaining about not being able to reach the target host and successfully did its job. I'm sure you'd like to see more logs but I'm not certain what exactly is relevant. There are a ton of logs as this deployment is comprised of nearly 80 hosts. So, I guess it's best if you just request to see specific logs, messages or configuration details and I'll cherry-pick what's relevant. We don't really understand what's going on and would appreciate any help. We tried just about anything we could think of to resolve this issue and are running out of ideas what to do next. If you have any questions just ask and I'll do my best to answer them.

2 4

Re: Failed to synchronize networks of Provider ovirt-provider-ovn
by Mail SET Inc. Group 27 Oct '23

27 Oct '23

Yes, i use same manual to change WebUI SSL. ovirt-ca-file= is a same SSL file which use WebUI. Yes, i restart ovirt-provider-ovn, i restart engine, i restart all what i can restart. Nothing... > 12 сент. 2018 г., в 16:11, Dominik Holler <dholler(a)redhat.com> написал(а): > > On Wed, 12 Sep 2018 14:23:54 +0300 > "Mail SET Inc. Group" <mail(a)set-pro.net> wrote: > >> Ok! > > Not exactly, please use users(a)ovirt.org for such questions. > Other should benefit from this questions, too. > Please write the next mail to users(a)ovirt.org and keep me in CC. > >> What i did: >> >> 1) install oVirt «from box» (4.2.5.2-1.el7); >> 2) generate own ssl for my engine using my FreeIPA CA, Install it and > > What means "Install it"? You can use the doc from the following link > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/ht… > > Ensure that ovirt-ca-file= in > /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf > points to the correct file and ovirt-provider-ovn is restarted. > >> get tis issue; >> >> >> [root@engine ~]# tail -n 50 /var/log/ovirt-provider-ovn.log >> 2018-09-12 14:10:23,828 root [SSL: CERTIFICATE_VERIFY_FAILED] >> certificate verify failed (_ssl.c:579) Traceback (most recent call >> last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", >> line 133, in _handle_request method, path_parts, content >> File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", >> line 175, in handle_request return >> self.call_response_handler(handler, content, parameters) File >> "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in >> call_response_handler return response_handler(content, parameters) >> File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", >> line 62, in post_tokens user_password=user_password) File >> "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in >> create_token return auth.core.plugin.create_token(user_at_domain, >> user_password) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line >> 48, in create_token timeout=self._timeout()) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, >> in create_token username, password, engine_url, ca_file, timeout) >> File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line >> 91, in _get_sso_token timeout=timeout File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, >> in wrapper response = func(*args, **kwargs) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, >> in wrapper raise BadGateway(e) BadGateway: [SSL: >> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) >> >> >> [root@engine ~]# tail -n 20 /var/log/ovirt-engine/engine.log >> 2018-09-12 14:10:23,773+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock >> Acquired to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:10:23,778+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] >> Running command: SyncNetworkProviderCommand internal: true. >> 2018-09-12 14:10:23,836+03 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] >> Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:10:23,837+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock >> freed to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:14:12,477+03 INFO >> [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default >> task-6) [] User admin@internal successfully logged in with scopes: >> ovirt-app-admin ovirt-app-api ovirt-app-portal >> ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all >> ovirt-ext=token-info:authz-search >> ovirt-ext=token-info:public-authz-search >> ovirt-ext=token-info:validate ovirt-ext=token:password-access >> 2018-09-12 14:14:12,587+03 INFO >> [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default >> task-6) [1bf1b763] Running command: CreateUserSessionCommand >> internal: false. 2018-09-12 14:14:12,628+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-6) [1bf1b763] EVENT_ID: USER_VDC_LOGIN(30), User >> admin@internal-authz connecting from '10.0.3.61' using session >> 's8jAm7BUJGlicthm6yZBA3CUM8QpRdtwFaK3M/IppfhB3fHFB9gmNf0cAlbl1xIhcJ2WX+ww7e71Ri+MxJSsIg==' >> logged in. 2018-09-12 14:14:30,972+03 INFO >> [org.ovirt.engine.core.bll.provider.ImportProviderCertificateCommand] >> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] Running >> command: ImportProviderCertificateCommand internal: false. Entities >> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: >> SystemAction group CREATE_STORAGE_POOL with role type ADMIN >> 2018-09-12 14:14:30,982+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] EVENT_ID: >> PROVIDER_CERTIFICATE_IMPORTED(213), Certificate for provider >> ovirt-provider-ovn was imported. (User: admin@internal-authz) >> 2018-09-12 14:14:31,006+03 INFO >> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] >> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Running >> command: TestProviderConnectivityCommand internal: false. Entities >> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: >> SystemAction group CREATE_STORAGE_POOL with role type ADMIN >> 2018-09-12 14:14:31,058+03 ERROR >> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] >> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Command >> 'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'default' is using 0 threads out of 1, 5 threads waiting for >> tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engine' is using 0 threads out of 500, 16 threads waiting for >> tasks and 0 tasks in queue. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engineScheduled' is using 0 threads out of 100, 100 threads >> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads >> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'hostUpdatesChecker' is using 0 threads out of 5, 2 threads >> waiting for tasks. 2018-09-12 14:15:23,843+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock >> Acquired to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:15:23,849+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] >> Running command: SyncNetworkProviderCommand internal: true. >> 2018-09-12 14:15:23,900+03 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] >> Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:15:23,901+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock >> freed to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' >> >> >> [root@engine ~]# >> cat /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf # >> This file is automatically generated by engine-setup. Please do not >> edit manually [OVN REMOTE] ovn-remote=ssl:127.0.0.1:6641 >> [SSL] >> https-enabled=true >> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem >> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer >> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass >> [OVIRT] >> ovirt-sso-client-secret=Ms7Gw9qNT6IkXu7oA54tDmxaZDIukABV >> ovirt-host=https://engine.set.local:443 >> ovirt-sso-client-id=ovirt-provider-ovn >> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem >> [PROVIDER] >> provider-host=engine.set.local >> >> >>> 12 сент. 2018 г., в 13:59, Dominik Holler <dholler(a)redhat.com> >>> написал(а): >>> >>> On Wed, 12 Sep 2018 13:04:53 +0300 >>> "Mail SET Inc. Group" <mail(a)set-pro.net> wrote: >>> >>>> Hello Dominik! >>>> I have a same issue with OVN provider and SSL >>>> https://www.mail-archive.com/users@ovirt.org/msg47020.html >>>> <https://www.mail-archive.com/users@ovirt.org/msg47020.html> But >>>> certificate changes not helps to resolve it. Maybe you can help me >>>> with this? >>> >>> Sure. Can you please share the relevant lines of >>> ovirt-provider-ovn.log and engine.log, and the information if you >>> are using the certificates generated by engine-setup with >>> users(a)ovirt.org ? Thanks, >>> Dominik >>> >> > >

5 13

Unable to enable HPET component of an specific VM in oVirt 4.7
by ricardoot＠gmail.com 24 Oct '23

24 Oct '23

Hello community members, I'm currently using oVirt 4.7 as my virtualization environment, and I'm facing an issue with enabling the HPET (High Precision Event Timer) component in the XML configuration file of virtual machine (VM). Upon inspecting the XML file, I noticed that the `<timer name='hpet' present='no'/>` line is missing, indicating that the HPET component is disabled. Here are the steps I have taken so far: 1. I verified that the VM's XML configuration file does not include the `<timer name='hpet' present='yes'/>` line. 2. While the VM was powered on, I used the following command to edit the XML configuration file: ``` virsh edit VM_NAME ``` I added the `<timer name='hpet' present='yes'/>` line to the XML file. However, the changes did not persist after restarting the VM. To provide additional information, on the host where oVirt is running, the available clock sources can be viewed by executing the following command: ``` cat /sys/devices/system/clocksource/clocksource0/available_clocksource ``` The output shows the available clock sources, such as `tsc`, `hpet`, and `acpi_pm`. To resolve the authentication issue with the `virsh` command, I created a user with appropriate privileges using the following command: ``` sudo saslpasswd2 -a libvirt USERNAME ``` After creating the user, I was able to authenticate successfully with the `virsh` command using the newly created credentials. However, I'm unable to find an option to add the HPET parameter in the web console of oVirt. It seems that the option to configure HPET is not available in the web console. Has anyone else encountered a similar issue in oVirt 4.7? Could you please provide guidance or suggest a solution to enable the HPET component in the XML configuration file of a powered-off VM in oVirt 4.7? Any insights, experiences, or suggestions would be greatly appreciated. Thank you in advance for your assistance! Best regards,

2 4

What is the status of the whole Ovirt Project?
by Arman Khalatyan 15 Oct '23

15 Oct '23

Hello everybody, What is the status of the ovirt project, would be continued it in the Rocky Linunx9? the last message is so sad: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/HEKKBM6MZEKBEA… Any good news/progress there? As a happy ovirt user since 2016 should we move to an another system? Thank you beforehand, Arman.

9 14

engine-config -s UserSessionTimeOutInterval=X problem
by marek 28 Aug '23

28 Aug '23

ovirt 4.5.4, standalone engine, centos 8 stream [root@ovirt ~]# engine-config -g UserSessionTimeOutInterval Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false UserSessionTimeOutInterval: 30 version: general [root@ovirt ~]# engine-config -s UserSessionTimeOutInterval=60 Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false Cannot set value 60 to key UserSessionTimeOutInterval. any ideas where is the problem? Marek

5 6

ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials
by cmicarelli＠fdmservices.it 22 Aug '23

22 Aug '23

Hi Guys, I'm tried to install ovirt 4.5 on centos 8 stream and everytime file with this error ovirt.hosted_engine_setup : Obtain SSO token using username/password credentials Someone could help me? Regrards

2 2

How restore nodes ovirt UP from NonResponsive and VMs executing
by José Pascual 08 Aug '23

08 Aug '23

Hello, I have ovirt with two nodes that are NonResponsive and all the VMs are properly executing but i cant manage them because are in Unknown state. It seems that nodes lost connection for a while with their gateway. I have thought of first restarting the node where the engine is not running and trying to put in UP. Then restart the engine from within de VM to see if it starts up on this node. What is the proper way of restoring management? I Thanks, Best Regards -- Saludos, José Pascual Gallud Martínez Nombre | Dpto. Ingeniería <http://telfy.com/>

3 2

Installing oVirt as a self-hosted engine - big big problem :()
by Jorge Visentini 03 Aug '23

03 Aug '23

Hi guys, starting the weekend with a "cucumber" like that in my hands. I've been racking my brains for about 4 days to deploy a new engine. Turns out I already tested *4.4.10*, *4.5.4.x*, and *4.5.5(master)* (el8 and el9) and none is working. It seems to me to be ansible or a python problem, but I'm not sure. I've read several oVirt reddit and github threads, but they seem to have no effect anymore. I believe it's some package in the CentOS Stream repositories, *but unfortunately I don't have it frozen locally here*. Deploy hangs at *[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Wait for the host to be up]* I already tried to update the version of *python netaddr* as I read in git and it still didn't work I also tried to *freeze the ansible update* in the engine and it didn't work. I updated the version of *ovirt-ansible-collection to ovirt-ansible-collection-3.1.3-0.1.master.20230420113738.el8.noarch.rpm* and it didn't work either... *The error seems to be on all oVirt builds*, but I don't know what I'm doing wrong anymore because I can't pinpoint where the error is. I appreciate any tips *Below are some log outputs:* [root@ksmmi1r02ovirt36 ~]# tail -f /var/log/vdsm/vdsm.log 2023-07-07 22:23:30,144-0300 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=6df1f5ed-0f41-4001-bb2e-e50fb0214ac7 (api:37) 2023-07-07 22:23:30,144-0300 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:723) 2023-07-07 22:23:35,146-0300 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList() from=internal, task_id=bd2a755d-3488-4b43-8ca4-44717dd6b017 (api:31) 2023-07-07 22:23:35,146-0300 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=bd2a755d-3488-4b43-8ca4-44717dd6b017 (api:37) 2023-07-07 22:23:35,146-0300 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:723) 2023-07-07 22:23:39,320-0300 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=68567ce3-b579-469d-a46d-7bafc7b3e6bd (api:31) 2023-07-07 22:23:39,320-0300 INFO (periodic/3) [vdsm.api] FINISH repoStats return={} from=internal, task_id=68567ce3-b579-469d-a46d-7bafc7b3e6bd (api:37) 2023-07-07 22:23:40,151-0300 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList() from=internal, task_id=fadcf734-9f7e-4681-8764-9d3863718644 (api:31) 2023-07-07 22:23:40,151-0300 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=fadcf734-9f7e-4681-8764-9d3863718644 (api:37) 2023-07-07 22:23:40,151-0300 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:723) 2023-07-07 22:23:44,183-0300 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,49920 (api:31) 2023-07-07 22:23:44,184-0300 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,49920 (api:37) 2023-07-07 22:23:45,157-0300 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList() from=internal, task_id=504d8028-35be-45a3-b24d-4ec7cbc82f7e (api:31) 2023-07-07 22:23:45,157-0300 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=504d8028-35be-45a3-b24d-4ec7cbc82f7e (api:37) 2023-07-07 22:23:45,157-0300 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:723) 2023-07-07 22:23:50,162-0300 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList() from=internal, task_id=297ad1df-c855-4fbb-a89f-dfbe7a1b60a2 (api:31) 2023-07-07 22:23:50,162-0300 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=297ad1df-c855-4fbb-a89f-dfbe7a1b60a2 (api:37) 2023-07-07 22:23:50,162-0300 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:723) [root@ksmmi1r02ovirt36 ~]# journalctl -f -- Logs begin at Fri 2023-07-07 21:57:13 -03. -- Jul 07 22:24:46 ksmmi1r02ovirt36.kosmo.cloud ansible-async_wrapper.py[13790]: 13791 still running (86045) Jul 07 22:24:50 ksmmi1r02ovirt36.kosmo.cloud platform-python[22812]: ansible-ovirt_host_info Invoked with pattern=name=ksmmi1r02ovirt36.kosmo.cloud auth={'token': 'eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyZzVfWUdWX08wSFJoWnlVeFNkdGl4d0liRWV6Wkp5NTgwN3BaXzUxelBvIn0.eyJleHAiOjE2ODg3OTY0MzAsImlhdCI6MTY4ODc3OTE1MCwianRpIjoiOTBlOWE5ZTAtMmExNS00MzNiLWIxOGQtMmUwNmI4MTQ5NGE2IiwiaXNzIjoiaHR0cHM6Ly9rc21lbmdpbmUwMS5rb3Ntby5jbG91ZC9vdmlydC1lbmdpbmUtYXV0aC9yZWFsbXMvb3ZpcnQtaW50ZXJuYWwiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYWRkMWMyYzYtYzJjMy00N2M4LWI1ODUtNGI2MTU2ZDAxYTE3IiwidHlwIjoiQmVhcmVyIiwiYXpwIjoib3ZpcnQtZW5naW5lLWludGVybmFsIiwic2Vzc2lvbl9zdGF0ZSI6IjNhMWUzZTU2LWIyZTUtNGMyYi05YTIxLThjZjE1YzY4NzlmNiIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cHM6Ly9rc21lbmdpbmUwMS5rb3Ntby5jbG91ZCJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1vdmlydC1pbnRlcm5hbCIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im92aXJ0LWV4dD10b2tlbjpwYXNzd29yZC1hY2Nlc3Mgb3ZpcnQtZXh0PXRva2VuLWluZm86cHVibGljLWF1dGh6LXNlYXJjaCBvdmlydC1hcHAtYXBpIG92aXJ0LWV4dD10b2tlbi1pbmZvOnZhbGlkYXRlIHByb2ZpbGUgZW1haWwgb3ZpcnQtZXh0PXRva2VuLWluZm86YXV0aHotc2VhcmNoIiwic2lkIjoiM2ExZTNlNTYtYjJlNS00YzJiLTlhMjEtOGNmMTVjNjg3OWY2IiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJncm91cHMiOlsiL292aXJ0LWFkbWluaXN0cmF0b3IiXSwicHJlZmVycmVkX3VzZXJuYW1lIjoiYWRtaW5Ab3ZpcnQiLCJlbWFpbCI6ImFkbWluQGxvY2FsaG9zdCJ9.o9PsulNw0urPphWITcB6Y3wpHQiiQ0v00su6XorITcvNElzkfHqyYfJd8W-kIfgElh6BNnCmYyIwtX7t3T4-PiLgDdipH1J9uzuDBXkmNBNcVmFimfUAqyC8aUITK56CqZ5TyRyHqhOicPciqGSY8R98hQ8I8y11w2RiIFT0rQYnRev75gjKoqUH29uNyeCAdTyKvPSGHNm1pLLrtPUmk-JCGmsYytNRCMHAPoNIlZP3k94PbQ9pI4jZ5O7kcRSgJik8tUDOVglcL4g0MoAJwracek2MUTvK8pDpRghI9hSQVLFtAXCyGRxfHHzTko4EbHBbFlz5s3pfs2kbF6TFmw', 'url': 'https://ksmengine01.kosmo.cloud/ovirt-engine/api', 'ca_file': None, 'insecure': True, 'timeout': 0, 'compress': True, 'kerberos': False, 'headers': None, 'hostname': None, 'username': None, 'password': None} fetch_nested=False nested_attributes=[] follow=[] all_content=False cluster_version=None Jul 07 22:24:51 ksmmi1r02ovirt36.kosmo.cloud ansible-async_wrapper.py[13790]: 13791 still running (86040) Jul 07 22:24:56 ksmmi1r02ovirt36.kosmo.cloud ansible-async_wrapper.py[13790]: 13791 still running (86035) Jul 07 22:25:00 ksmmi1r02ovirt36.kosmo.cloud platform-python[22829]: ansible-ovirt_host_info Invoked with pattern=name=ksmmi1r02ovirt36.kosmo.cloud auth={'token': 'eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIyZzVfWUdWX08wSFJoWnlVeFNkdGl4d0liRWV6Wkp5NTgwN3BaXzUxelBvIn0.eyJleHAiOjE2ODg3OTY0MzAsImlhdCI6MTY4ODc3OTE1MCwianRpIjoiOTBlOWE5ZTAtMmExNS00MzNiLWIxOGQtMmUwNmI4MTQ5NGE2IiwiaXNzIjoiaHR0cHM6Ly9rc21lbmdpbmUwMS5rb3Ntby5jbG91ZC9vdmlydC1lbmdpbmUtYXV0aC9yZWFsbXMvb3ZpcnQtaW50ZXJuYWwiLCJhdWQiOiJhY2NvdW50Iiwic3ViIjoiYWRkMWMyYzYtYzJjMy00N2M4LWI1ODUtNGI2MTU2ZDAxYTE3IiwidHlwIjoiQmVhcmVyIiwiYXpwIjoib3ZpcnQtZW5naW5lLWludGVybmFsIiwic2Vzc2lvbl9zdGF0ZSI6IjNhMWUzZTU2LWIyZTUtNGMyYi05YTIxLThjZjE1YzY4NzlmNiIsImFjciI6IjEiLCJhbGxvd2VkLW9yaWdpbnMiOlsiaHR0cHM6Ly9rc21lbmdpbmUwMS5rb3Ntby5jbG91ZCJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiZGVmYXVsdC1yb2xlcy1vdmlydC1pbnRlcm5hbCIsIm9mZmxpbmVfYWNjZXNzIiwidW1hX2F1dGhvcml6YXRpb24iXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im92aXJ0LWV4dD10b2tlbjpwYXNzd29yZC1hY2Nlc3Mgb3ZpcnQtZXh0PXRva2VuLWluZm86cHVibGljLWF1dGh6LXNlYXJjaCBvdmlydC1hcHAtYXBpIG92aXJ0LWV4dD10b2tlbi1pbmZvOnZhbGlkYXRlIHByb2ZpbGUgZW1haWwgb3ZpcnQtZXh0PXRva2VuLWluZm86YXV0aHotc2VhcmNoIiwic2lkIjoiM2ExZTNlNTYtYjJlNS00YzJiLTlhMjEtOGNmMTVjNjg3OWY2IiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJncm91cHMiOlsiL292aXJ0LWFkbWluaXN0cmF0b3IiXSwicHJlZmVycmVkX3VzZXJuYW1lIjoiYWRtaW5Ab3ZpcnQiLCJlbWFpbCI6ImFkbWluQGxvY2FsaG9zdCJ9.o9PsulNw0urPphWITcB6Y3wpHQiiQ0v00su6XorITcvNElzkfHqyYfJd8W-kIfgElh6BNnCmYyIwtX7t3T4-PiLgDdipH1J9uzuDBXkmNBNcVmFimfUAqyC8aUITK56CqZ5TyRyHqhOicPciqGSY8R98hQ8I8y11w2RiIFT0rQYnRev75gjKoqUH29uNyeCAdTyKvPSGHNm1pLLrtPUmk-JCGmsYytNRCMHAPoNIlZP3k94PbQ9pI4jZ5O7kcRSgJik8tUDOVglcL4g0MoAJwracek2MUTvK8pDpRghI9hSQVLFtAXCyGRxfHHzTko4EbHBbFlz5s3pfs2kbF6TFmw', 'url': 'https://ksmengine01.kosmo.cloud/ovirt-engine/api', 'ca_file': None, 'insecure': True, 'timeout': 0, 'compress': True, 'kerberos': False, 'headers': None, 'hostname': None, 'username': None, 'password': None} fetch_nested=False nested_attributes=[] follow=[] all_content=False cluster_version=None cat /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230707220613-2t8ze9.log 2023-07-07 22:19:11,816-0300 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Wait for the host to be up] 2023-07-07 22:39:39,882-0300 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'changed': False, 'ovirt_hosts': [{'href': '/ovirt-engine/api/hosts/d1bf8fb2-74f4-4954-8c34-66eb99ba2bf3', 'comment': '', 'id': 'd1bf8fb2-74f4-4954-8c34-66eb99ba2bf3', 'name': 'ksmmi1r02ovirt36.kosmo.cloud', 'address': 'ksmmi1r02ovirt36.kosmo.cloud', 'affinity_labels': [], 'auto_numa_status': 'unknown', 'certificate': {'organization': 'kosmo.cloud', 'subject': 'O=kosmo.cloud,CN=ksmmi1r02ovirt36.kosmo.cloud'}, 'cluster': {'href': '/ovirt-engine/api/clusters/d8784faf-8b77-45c8-9fa4-b9b4b0404d95', 'id': 'd8784faf-8b77-45c8-9fa4-b9b4b0404d95'}, 'cpu': {'speed': 0.0, 'topology': {}}, 'cpu_units': [], 'device_passthrough': {'enabled': False}, 'devices': [], 'external_network_provider_configurations': [], 'external_status': 'ok', 'hardware_information': {'supported_rng_sources': []}, 'hooks': [], 'katello_errata': [], 'kdump_status': 'unknown', 'ksm': {'enabled': False}, 'max_scheduling_memory': 0, 'memory': 0, 'network_attachments': [], 'nics': [], 'numa_nodes': [], 'numa_supported': False, 'os': {'custom_kernel_cmdline': ''}, 'ovn_configured': False, 'permissions': [], 'port': 54321, 'power_management': {'automatic_pm_enabled': True, 'enabled': False, 'kdump_detection': True, 'pm_proxies': []}, 'protocol': 'stomp', 'reinstallation_required': False, 'se_**FILTERED**': {}, 'spm': {'priority': 5, 'status': 'none'}, 'ssh': {'fingerprint': 'SHA256:Nr04m1g0UxbpqxwMBr93DLHz2m2wzR8+xFJhBVNovHY', 'port': 22, 'public_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2EdJn0vJiJUagEK3w2G2nHmziJJasailwapaL06qWU2+BkPwkokSvyK07APhwyynnz6lw8J4y/kWv12D7/r+s='}, 'statistics': [], 'status': 'install_failed', 'storage_connection_extensions': [], 'summary': {'total': 0}, 'tags': [], 'transparent_huge_pages': {'enabled': False}, 'type': 'rhel', 'unmanaged_networks': [], 'update_available': False, 'vgpu_placement': 'consolidated'}], 'invocation': {'module_args': {'pattern': 'name=ksmmi1r02ovirt36.kosmo.cloud', 'fetch_nested': False, 'nested_attributes': [], 'follow': [], 'all_content': False, 'cluster_version': None}}, '_ansible_no_log': None, 'attempts': 120} 2023-07-07 22:39:39,983-0300 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ignored: [localhost]: FAILED! => {"attempts": 120, "changed": false, "ovirt_hosts": [{"address": "ksmmi1r02ovirt36.kosmo.cloud", "affinity_labels": [], "auto_numa_status": "unknown", "certificate": {"organization": "kosmo.cloud", "subject": "O=kosmo.cloud,CN=ksmmi1r02ovirt36.kosmo.cloud"}, "cluster": {"href": "/ovirt-engine/api/clusters/d8784faf-8b77-45c8-9fa4-b9b4b0404d95", "id": "d8784faf-8b77-45c8-9fa4-b9b4b0404d95"}, "comment": "", "cpu": {"speed": 0.0, "topology": {}}, "cpu_units": [], "device_passthrough": {"enabled": false}, "devices": [], "external_network_provider_configurations": [], "external_status": "ok", "hardware_information": {"supported_rng_sources": []}, "hooks": [], "href": "/ovirt-engine/api/hosts/d1bf8fb2-74f4-4954-8c34-66eb99ba2bf3", "id": "d1bf8fb2-74f4-4954-8c34-66eb99ba2bf3", "katello_errata": [], "kdump_status": "unknown", "ksm": {"enabled": false}, "max_scheduling_memory": 0, "memory": 0, "name": "ksmmi1r02ovirt36.kosmo.cloud", "network_attachments": [], "nics": [], "numa_nodes": [], "numa_supported": false, "os": {"custom_kernel_cmdline": ""}, "ovn_configured": false, "permissions": [], "port": 54321, "power_management": {"automatic_pm_enabled": true, "enabled": false, "kdump_detection": true, "pm_proxies": []}, "protocol": "stomp", "reinstallation_required": false, "se_**FILTERED**": {}, "spm": {"priority": 5, "status": "none"}, "ssh": {"fingerprint": "SHA256:Nr04m1g0UxbpqxwMBr93DLHz2m2wzR8+xFJhBVNovHY", "port": 22, "public_key": "ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE2EdJn0vJiJUagEK3w2G2nHmziJJasailwapaL06qWU2+BkPwkokSvyK07APhwyynnz6lw8J4y/kWv12D7/r+s="}, "statistics": [], "status": "install_failed", "storage_connection_extensions": [], "summary": {"total": 0}, "tags": [], "transparent_huge_pages": {"enabled": false}, "type": "rhel", "unmanaged_networks": [], "update_available": false, "vgpu_placement": "consolidated"}]} 2023-07-07 22:39:40,284-0300 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about a failure] 2023-07-07 22:39:40,685-0300 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'msg': 'Host is not up, please check logs, perhaps also on the engine machine', '_ansible_no_log': None, 'changed': False} 2023-07-07 22:39:40,786-0300 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Host is not up, please check logs, perhaps also on the engine machine"} Have a nice weekend! -- Att, Jorge Visentini +55 55 98432-9868

2 2