- Users - oVirt List Archives

Unable to edit DNS connfiguration on hosts
by miguel.garcia＠toshibagcs.com 20 Jan '21

20 Jan '21

I had to replace the DNS server in my network so I want to update the DNS address on each host so in oVirt web interface I go to Compute > Host > Hostname > Network Interfaces > Setup Host Networks Right there I click on the edit button of the vlan for networking, check out (host ip address is static), DNS configuration, check DNS configuration, Type DNS ip address and click ok to save change. I got message: Error while executing action HostSetupNetworks: Unexpected Exception. Into engile.log found message "Operation not permitted" related to this action despite that was executed with my user account (wint admin privileges) and same message was returned using admin@internal user. 2021-01-20 11:35:31,579-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] START, HostSetupNetworksVDSCommand(HostName = hyp10.eboss, HostSetupNetworksVdsCommandParameters:{hostId='39c05625-7b59-4a38-a5a9-5d556ae1fe18', vds='Host[hyp10.eboss,39c05625-7b59-4a38-a5a9-5d556ae1fe18]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='tgcs182vlan', vdsmName='tgcs182vlan', nicName='eno1', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STATIC_IP', ipv4Address='10.89.182.10', ipv4Netmask='255.255.255.0', ipv4Gateway='10.89.182.1', ipv6BootProtocol='AUTOCONF', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='[NameServer:{address='10.89.182.204'}, NameServer:{address='10.89.176.71'}]'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSw itchType='OVS', managementNetworkChanged='false'}), log id: 23059843 2021-01-20 11:35:31,582-05 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] FINISH, HostSetupNetworksVDSCommand, return: , log id: 23059843 2021-01-20 11:35:35,665-05 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': '[Errno 1] Operation not permitted'}] 2021-01-20 11:35:35,665-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] Failed in 'HostSetupNetworksVDS' method 2021-01-20 11:35:35,666-05 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': '[Errno 1] Operation not permitted'}] 2021-01-20 11:35:35,673-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM hyp10.eboss command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': '[Errno 1] Operation not permitted'} 2021-01-20 11:35:35,673-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] Error: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': '[Errno 1] Operation not permitted'}, code = -32603 2021-01-20 11:35:35,673-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] Command 'HostSetupNetworksVDSCommand(HostName = hyp10.eboss, HostSetupNetworksVdsCommandParameters:{hostId='39c05625-7b59-4a38-a5a9-5d556ae1fe18', vds='Host[hyp10.eboss,39c05625-7b59-4a38-a5a9-5d556ae1fe18]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='tgcs182vlan', vdsmName='tgcs182vlan', nicName='eno1', vlan='null', vmNetwork='true', stp='false', properties='[]', ipv4BootProtocol='STATIC_IP', ipv4Address='10.89.182.10', ipv4Netmask='255.255.255.0', ipv4Gateway='10.89.182.1', ipv6BootProtocol='AUTOCONF', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='[NameServer:{address='10.89.182.204'}, NameServer:{address='10.89.176.71'}]'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', cluster SwitchType='OVS', managementNetworkChanged='false'})' execution failed: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': '[Errno 1] Operation not permitted'}, code = -32603 2021-01-20 11:35:35,673-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] Host setup networks finished. Lock released. Monitoring can run now for host 'hyp10.eboss' from data-center 'tgcs-gdl-dc1-public' 2021-01-20 11:35:35,673-05 ERROR [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] Command 'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': '[Errno 1] Operation not permitted'}, code = -32603 (Failed with error unexpected and code 16) 2021-01-20 11:35:35,677-05 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (default task-482931) [b9a19834-3f15-42da-af2c-55e292dfedea] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK39c05625-7b59-4a38-a5a9-5d556ae1fe18=HOST_NETWORK]', sharedLocks=''}' This error message happens even in master as in nodes hosts. Any idea how can I solve it? Thanks in advance

1 0

About Enroll Certificate
by tommy 20 Jan '21

20 Jan '21

Hi, What function of the Enroll Certificate ??

2 2

Problems Installing host on brand new 4.4 cluster
by David Johnson 20 Jan '21

20 Jan '21

Hi all, I am standing up a brand new cluster on new hardware. The ovirt controller is installed and appears to be running fine. When I attempt to add the new host, I get the flag that says "Non Operational", and the exclamation mark with the message "Host has no default route". I have confirmed that I have bidirectional ssh connections between the controller and the new host. Selinux is disabled. Firewall is disabled. I can get to the host console via the link in the GUI Compute|Hosts page. This looks like a common symptom of a host of problems, but there is nothing to readily indicate what the actual problem is. In the ovirt-engine log, I found this entry: "Host 'ovirt-host-03' is set to Non-Operational, it is missing the following networks: 'ovirtmgmt'" This seems self-explanatory, but I see no way to add the missing network to the host. Thank you in advance for your help Regards, David Johnson * <http://www.linkedin.com/in/pojoguy>*

2 2

Install of RHV 4.4 failing - "Host is not up, please check logs, perhaps also on the engine machine"
by james.freeman＠a24.io 19 Jan '21

19 Jan '21

Hi all I am in the process of migrating a RHV 4.3 setup to RHV 4.4 and struggling with the setup. I am installing on RHEL 8.3, using settings backed up from the RHV 4.3 install (via 'hosted-engine --deploy --restore-from-file=backup.bck'). The install process always fails at the same point for me at the moment, and I can't figure out how to get past it. As far as install progress goes, the local hosted-engine comes up and runs on the node. I have been able to grep for local_vm_ip in the logs, and can SSH into it with the password I set during the setup phase. However the install playbooks always fail with: 2021-01-18 18:38:00,086-0500 ERROR otopi.plugins.gr_he_common.core.misc misc._terminate:167 Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Earlier in the logs, I note the following: 2021-01-18 18:34:51,258-0500 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Host is not up, please check logs, perhaps also on the engine machine"} 2021-01-18 18:37:16,661-0500 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-ansiblesetup/core/misc.py", line 435, in _closeup raise RuntimeError(_('Failed executing ansible-playbook')) RuntimeError: Failed executing ansible-playbook 2021-01-18 18:37:18,996-0500 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': Failed executing ansible-playbook 2021-01-18 18:37:32,421-0500 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host rhvm.example.org port 22: No route to host", "skip_reason": "Host localhost is unreachable", "unreachable": true} I find the unreachable message a bit odd, as at this stage all that has happened is that the local hosted-engine has been brought up to be configured, and so it is running on virbr0, not on my actual network. As a result, that DNS address will never resolve, and the IP it resolves to won't be up. I gave the installation script permission to modify the local /etc/hosts but this hasn't improved things. I presume I'm missing something in the install process, or earlier on in the logs, but I've been scanning for errors and possible clues to no avail. Any and all help greatly appreciated!

3 5

VM Disks order
by Erez Zarum 19 Jan '21

19 Jan '21

When attaching a disk it is not possible to set the disk order nor modify the order later. Example: A new VM is provisioned with 5 disks, Disk0 is the OS and then later attached disks by order up to Disk4. Removing Disk3 and then later attaching does not promise it will be attached back as Disk3. In most other platforms it is possible to set the order. Am i missing something? if not, is there a plan to add this feature?

2 1

Why the update hung here ?
by tommy 19 Jan '21

19 Jan '21

Hi , I just update the hosts, but why the log print these info many many times for a long while.

2 1

noVNC error.
by tommy 19 Jan '21

19 Jan '21

Hi: I use novnc console can connect to engine vm. But when I using novnc console connect to other vm in other datacenter, failed. The ovirt-websocket-proxy log is: Jan 19 15:43:32 ooeng.tltd.com ovirt-websocket-proxy.py[1312]: 192.168.10.104 - - [19/Jan/2021 15:43:32] connecting to: ohost2.tltd.com:5900 (using SSL) Jan 19 15:43:32 ooeng.tltd.com ovirt-websocket-proxy.py[1312]: ovirt-websocket-proxy[24096] INFO msg:824 handler exception: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:618) What reason ?

2 1

Guest don't start : Cannot access backing file
by Lionel Caignec 19 Jan '21

19 Jan '21

Hi, i've a big problem, i juste shutdown (power off completely) a guest to make a cold restart. And at startup the guest say : "Cannot access backing file '/rhev/data-center/mnt/blockSD/69348aea-7f55-41be-ae4e-febd86c33855/images/8224b2b0-39ba-44ef-ae41-18fe726f26ca/ca141675-c6f5-4b03-98b0-0312254f91e8'" When i look from shell on Hypervisor the device file is red blinking... Trying to change SPM, look device into all hosts, copy disk etc... no way to get my disk back online. It seems ovirt completely lost (delete?) the block device. There is a way to manually dump (dd) the device in command line in order to import it inside ovirt? My environment Storage : SAN (managed by ovirt) Ovirt-engine 4.4.3.12-1.el8 Host centos 8.2 Vdsm 4.40.26 Thanks for help i'm stuck and it's really urgent -- Lionel Caignec

2 2

oVirt Engine no longer Starting
by penguin pages 18 Jan '21

18 Jan '21

3 node cluster. Gluster for shared storage. CentOS8 Updated to CentOS 8 Streams :P -> https://bugzilla.redhat.com/show_bug.cgi?id=1911910 After several weeks .. I am really in need of direction to get this fixed. I saw several postings about oVirt package issues but not found a fix. [root@thor ~]# dnf update Last metadata expiration check: 2:54:29 ago on Fri 15 Jan 2021 06:49:16 AM EST. Error: Problem 1: package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-bridge-234-1.el8.x86_64 conflicts with cockpit-dashboard < 233 provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package ovirt-host-4.4.1-4.el8.x86_64 - cannot install the best update candidate for package cockpit-bridge-217-1.el8.x86_64 Problem 2: problem with installed package ovirt-host-4.4.1-4.el8.x86_64 - package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package cockpit-dashboard-217-1.el8.noarch Problem 3: package ovirt-hosted-engine-setup-2.4.9-1.el8.noarch requires ovirt-host >= 4.4.0, but none of the providers can be installed - package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-1.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-2.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-3.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package ovirt-hosted-engine-setup-2.4.9-1.el8.noarch - cannot install the best update candidate for package cockpit-system-217-1.el8.noarch (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) [root@thor ~]# yum install cockpit-dashboard --nobest Last metadata expiration check: 2:54:52 ago on Fri 15 Jan 2021 06:49:16 AM EST. Package cockpit-dashboard-217-1.el8.noarch is already installed. Dependencies resolved. Problem: problem with installed package ovirt-host-4.4.1-4.el8.x86_64 - package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-1.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-2.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-3.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best candidate for the job ========================================================================================================================================================================================================================================= Package Architecture Version Repository Size ========================================================================================================================================================================================================================================= Skipping packages with broken dependencies: ovirt-host x86_64 4.4.1-1.el8 ovirt-4.4 13 k ovirt-host x86_64 4.4.1-2.el8 ovirt-4.4 13 k ovirt-host x86_64 4.4.1-3.el8 ovirt-4.4 13 k Transaction Summary ========================================================================================================================================================================================================================================= Skip 3 Packages Nothing to do. Complete! [root@thor ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 931.5G 0 disk └─WDC_WDS100T2B0B-00YS70_19106A802926 253:3 0 931.5G 0 mpath └─vdo_2926 253:5 0 4T 0 vdo /gluster_bricks/gv0 sdb 8:16 0 931.5G 0 disk └─WDC_WDS100T2B0B-00YS70_192490801828 253:4 0 931.5G 0 mpath sdc 8:32 0 477G 0 disk └─vdo_sdc 253:6 0 2.1T 0 vdo ├─gluster_vg_sdc-gluster_lv_engine 253:7 0 100G 0 lvm /gluster_bricks/engine ├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc_tmeta 253:8 0 1G 0 lvm │ └─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc-tpool 253:10 0 2T 0 lvm │ ├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc 253:11 0 2T 1 lvm │ ├─gluster_vg_sdc-gluster_lv_data 253:12 0 1000G 0 lvm /gluster_bricks/data │ └─gluster_vg_sdc-gluster_lv_vmstore 253:13 0 1000G 0 lvm /gluster_bricks/vmstore └─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc_tdata 253:9 0 2T 0 lvm └─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc-tpool 253:10 0 2T 0 lvm ├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc 253:11 0 2T 1 lvm ├─gluster_vg_sdc-gluster_lv_data 253:12 0 1000G 0 lvm /gluster_bricks/data └─gluster_vg_sdc-gluster_lv_vmstore 253:13 0 1000G 0 lvm /gluster_bricks/vmstore sdd 8:48 1 58.8G 0 disk ├─sdd1 8:49 1 1G 0 part /boot └─sdd2 8:50 1 57.8G 0 part ├─cl-root 253:0 0 36.1G 0 lvm / ├─cl-swap 253:1 0 4G 0 lvm [SWAP] └─cl-home 253:2 0 17.6G 0 lvm /home [root@thor ~]# mount |grep engine /dev/mapper/gluster_vg_sdc-gluster_lv_engine on /gluster_bricks/engine type xfs (rw,noatime,nodiratime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota,_netdev,x-systemd.requires=vdo.service) thorst.penguinpages.local:/engine on /media/engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072,_netdev) thorst.penguinpages.local:/engine on /rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072,_netdev,x-systemd.device-timeout=0) [root@thor ~]# [root@thor ~]# tail -50 /var/log/messages Jan 15 09:46:43 thor platform-python[28088]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:43 thor abrt-server[28116]: Not saving repeating crash in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Main process exited, code=exited, status=1/FAILURE Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Failed with result 'exit-code'. Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms expired, scheduling restart. Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart counter is at 241. Jan 15 09:46:43 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:43 thor systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:45 thor systemd[1]: Started Session c448 of user root. Jan 15 09:46:45 thor systemd[1]: session-c448.scope: Succeeded. Jan 15 09:46:45 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor] does not exist on server localhost Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Service RestartSec=10s expired, scheduling restart. Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Scheduled restart job, restart counter is at 211. Jan 15 09:46:48 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Jan 15 09:46:48 thor systemd[1]: Started oVirt Hosted Engine High Availability Monitoring Agent. Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Failed initializing the broker: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata' Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 64, in run#012 self._storage_broker_instance = self._get_storage_broker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 143, in _get_storage_broker#012 return storage_broker.StorageBroker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 97, in __init__#012 self._backend.connect()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 408, in connect#012 self._check_symlinks(self._storage_path, volume.path, service_link)#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 105, in _check_symlinks#012 os.unlink(service_link)#012OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata' Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Trying to restart the broker Jan 15 09:46:48 thor platform-python[28118]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Main process exited, code=exited, status=1/FAILURE Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Failed with result 'exit-code'. Jan 15 09:46:48 thor abrt-server[28144]: Deleting problem directory Python3-2021-01-15-09:46:48-28118 (dup of Python3-2020-09-18-14:25:13-1363) Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms expired, scheduling restart. Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart counter is at 242. Jan 15 09:46:48 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:48 thor systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 85, in start_monitor#012 response = self._proxy.start_monitor(type, options)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__#012 return self.__send(self.__name, args)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request#012 verbose=self.__verbose#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request#012 return self.single_request(host, handler, request_body, verbose)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in single_request#012 http_conn = self.send_request(host, handler, request_body, verbose)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in send_request#012 self.send_content(connection, request_body)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1309, in send_content#012 connection.endheaders(request_body)#012 File "/usr/lib64/python3.6/http/client.py", line 1264, in endheaders#012 self._send_output(message_body, encode_chunked=encode_chunked)#012 File "/usr/lib64/python3.6/http/client.py", line 1040, in _send_output#012 self.send(msg)#012 File "/usr/lib64/python3.6/http/client.py", line 978, in send#012 self.connect()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 74, in connect#012 self.sock.connect(base64.b16decode(self.host))#012FileNotFoundError: [Errno 2] No such file or directory#012#012During handling of the above exception, another exception occurred:#012#012Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 437, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 561, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 91, in start_monitor#012 ).format(t=type, o=options, e=e)#012ovirt_hosted_engine_ha.lib.exceptions.RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'addr': '172.16.100.1', 'network_test': 'dns', 'tcp_t_address': '', 'tcp_t_port': ''}] Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Jan 15 09:46:48 thor abrt-server[28144]: /bin/sh: reporter-systemd-journal: command not found Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Main process exited, code=exited, status=157/n/a Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Failed with result 'exit-code'. Jan 15 09:46:49 thor vdsm[8421]: WARN Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? Jan 15 09:46:50 thor systemd[1]: Started Session c449 of user root. Jan 15 09:46:50 thor systemd[1]: session-c449.scope: Succeeded. Jan 15 09:46:50 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor] does not exist on server localhost Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Failed initializing the broker: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata' Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 64, in run#012 self._storage_broker_instance = self._get_storage_broker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 143, in _get_storage_broker#012 return storage_broker.StorageBroker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 97, in __init__#012 self._backend.connect()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 408, in connect#012 self._check_symlinks(self._storage_path, volume.path, service_link)#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 105, in _check_symlinks#012 os.unlink(service_link)#012OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata' Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Trying to restart the broker Jan 15 09:46:53 thor platform-python[28165]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:53 thor abrt-server[28199]: Not saving repeating crash in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Main process exited, code=exited, status=1/FAILURE Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Failed with result 'exit-code'. Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms expired, scheduling restart. Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart counter is at 243. Jan 15 09:46:53 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:53 thor systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:55 thor systemd[1]: Started Session c450 of user root. Jan 15 09:46:55 thor systemd[1]: session-c450.scope: Succeeded. Jan 15 09:46:55 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor] does not exist on server localhost Questions: 1) I have two important VMs that have snapshots that I need to boot up. Is their a means with an HCI configuration to manually start the VMs without oVirt engine being up? 2) Is their a means to debug what is going on with the engine failing to start to repair (I hate reloading as the only fix for systems) 3) Is their a means to re-deploy HCI setup wizard, but use the "engine" volume and so retain the VMs and templates?

4 19

VG issue / Non Operational
by Christian Reiss 18 Jan '21

18 Jan '21

Hey folks, quick (I hope) question: On my 3-node cluster I am swapping out all the SSDs with fewer but higher capacity ones. So I took one node down (maintenance, stop), then removed all SSDs, set up a new RAID, set up lvm and gluster, let it resync. Gluster health status shows no unsynced entries. Uppon going from maintenance to online from ovirt management It goes into non-operational status, vdsm log on the node shows: 2021-01-17 11:13:29,051+0100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,48580 (api:48) 2021-01-17 11:13:29,051+0100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,48580 (api:54) 2021-01-17 11:13:29,052+0100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:312) 2021-01-17 11:13:30,420+0100 WARN (monitor/4a62cdb) [storage.LVM] Reloading VGs failed (vgs=[u'4a62cdb4-b314-4c7f-804e-8e7275518a7f'] rc=5 out=[] err=[' Volume group "4a62cdb4-b314-4c7f-804e-8e7275518a7f" not found', ' Cannot process volume group 4a62cdb4-b314-4c7f-804e-8e7275518a7f']) (lvm:470) 2021-01-17 11:13:30,424+0100 ERROR (monitor/4a62cdb) [storage.Monitor] Setting up monitor for 4a62cdb4-b314-4c7f-804e-8e7275518a7f failed (monitor:330) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 327, in _setupLoop self._setupMonitor() File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 349, in _setupMonitor self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 159, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python2.7/site-packages/vdsm/storage/monitor.py", line 367, in _produceDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 110, in produce domain.getRealDomain() File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 51, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 134, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 151, in _findDomain return findMethod(sdUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/sdc.py", line 176, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) I assume due to the changed LVM UUID it fails, right? Can I someone fix/change the UUID and get the node back up again? It does not seem to be a major issue, to be honest. I can see the gluster mount (what ovirt mounts when it onlines a node) already, and gluster is happy too. Any help is appreciated! -Chris. -- with kind regards, mit freundlichen Gruessen, Christian Reiss

2 4