- Users - oVirt List Archives

Gluster volume engine stuck in healing with 1 unsynched entry & HostedEngine paused
by souvaliotimaria＠mail.com 11 Mar '21

11 Mar '21

Hello everyone, Any help would be greatly appreciated in the following problem. In my lab, the day before yesterday, we had power issues, with a UPS going off-line and following the power outage of the NFS/DNS server I have set up to serve ovirt with isos and as a DNS server (our other DNS servers are located as VMs within the oVirt environment). We found a broadcast storm on the switch (due to a faulty NIC on the aformentioned UPS) that the ovirt nodes are connected and later on had to re-establish several of the virtual connections as well. The above led to one of the hosts becoming NonResponsive, two machines becoming unresponsive and three VMs shuting down. The oVirt environment, version 4.3.5.2, is a replica 2 + arbiter 1 environment and runs GlusterFS with the recommended volumes of data, engine and vmstore. So far, the times there was some kind of a problem, usually oVirt was able to solve it by its own. This time, however, after we recovered from the above state, the volumes of data and vmstore successfully healing , the volume engine became stuck to the healing process (Up, unsynched entries, needs healing), and from the web GUI I see that the VM HostedEngine is paused due to a storage I/O error while the output of virsh list --all command shows that the HostedEngine is running.. How is that happening? I tried to manually trigger the healing process for the volume but nothing with gluster volume heal engine The command gluster volume heal engine info shows the following [root@ov-no3 ~]# gluster volume heal engine info Brick ov-no1.ariadne-t.local:/gluster_bricks/engine/engine Status: Connected Number of entries: 0 Brick ov-no2.ariadne-t.local:/gluster_bricks/engine/engine /80f6e393-9718-4738-a14a-64cf43c3d8c2/images/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7 Status: Connected Number of entries: 1 Brick ov-no3.ariadne-t.local:/gluster_bricks/engine/engine /80f6e393-9718-4738-a14a-64cf43c3d8c2/images/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7 Status: Connected Number of entries: 1 This morning I came upon this Reddit post https://www.reddit.com/r/gluster/comments/fl3yb7/entries_stuck_in_heal_pend… where it seems that after a graceful reboot one of the ovirt hosts, the gluster came back online after it completed the appropriate healing processes. The thing is from what I have read that when there are unsynched entries in the gluster a host cannot be put into maintenance mode so that it can be rebooted, correct? Should I try to restart the glusterd service. Could someone tell me what I should do? Thank you all for your time and help, Maria Souvalioti

5 21

VM CPU Pinning
by lavi.buchnik＠exlibrisgroup.com 11 Mar '21

11 Mar '21

Hi, When trying to pin a VM CPU's to Physical CPU's, I'm getting this error in case the number of physical CPU slots are > 35 (e.g. 40): Error attempting to pin CPUs. Full error: Fault reason is "Operation Failed". Fault detail is "[size must be between 0 and 4000, Attribute: vmStatic.cpuPinning]". HTTP response code is "400". HTTP response message is "Bad Request". It is working for me up to 35 physical CPU's. but above it, I get that error. Can you please tell what this error means and how to overcome it? Version we are using is: 4.3.6.6-1.0.9.el7 Thanks, Lavi

2 4

Update Package Conflict
by penguin pages 11 Mar '21

11 Mar '21

Fresh install of minimal CentOS8 Then deploy: - EPEL - Add ovirt repo https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm Install all nodes: - cockpit-ovirt-dashboard - gluster-ansible-roles - vdsm-gluster - ovirt-host - ovirt-ansible-roles - ovirt-ansible-infra Install on "first node of cluster" - ovirt-engine-appliance Now each node is stuck with same package conflict error: (and this blocks GUI "upgrades") [root@medusa ~]# yum update Last metadata expiration check: 0:55:35 ago on Wed 10 Mar 2021 08:14:22 AM EST. Error: Problem 1: package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-bridge-238.1-1.el8.x86_64 conflicts with cockpit-dashboard < 233 provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package ovirt-host-4.4.1-4.el8.x86_64 - cannot install the best update candidate for package cockpit-bridge-217-1.el8.x86_64 Problem 2: problem with installed package ovirt-host-4.4.1-4.el8.x86_64 - package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-system-238.1-1.el8.noarch obsoletes cockpit-dashboard provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package cockpit-dashboard-217-1.el8.noarch Problem 3: package ovirt-hosted-engine-setup-2.4.9-1.el8.noarch requires ovirt-host >= 4.4.0, but none of the providers can be installed - package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-1.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-2.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-3.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-system-238.1-1.el8.noarch obsoletes cockpit-dashboard provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package ovirt-hosted-engine-setup-2.4.9-1.el8.noarch - cannot install the best update candidate for package cockpit-system-217-1.el8.noarch (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) [root@medusa ~]# yum update --allowerasing Last metadata expiration check: 0:55:56 ago on Wed 10 Mar 2021 08:14:22 AM EST. Dependencies resolved. ========================================================================================================================================================================================================================================= Package Architecture Version Repository Size ========================================================================================================================================================================================================================================= Upgrading: cockpit-bridge x86_64 238.1-1.el8 baseos 535 k cockpit-system noarch 238.1-1.el8 baseos 3.4 M replacing cockpit-dashboard.noarch 217-1.el8 Removing dependent packages: cockpit-ovirt-dashboard noarch 0.14.17-1.el8 @ovirt-4.4 16 M ovirt-host x86_64 4.4.1-4.el8 @ovirt-4.4 11 k ovirt-hosted-engine-setup noarch 2.4.9-1.el8 @ovirt-4.4 1.3 M Transaction Summary ========================================================================================================================================================================================================================================= Upgrade 2 Packages Remove 3 Packages ## Initially I assumed this was a path I was taking that was not standard.. but now I think this is some ovirt vs CentOS package repo issue. Any work arounds or root cause to fix this from repo conflict?

3 6

How to replace a failed oVirt Hyperconverged Host
by Ramon Sierra 10 Mar '21

10 Mar '21

Hi, We have a three hosts hyperconverged ovirt setup. A few weeks ago one of the hosts failed and we lost a RAID5 array on it. We removed it from the cluster and repaired. We are trying to setup and add it back to the cluster but we are not clear on how to proceed. There is a replica 2 with 1 arbiter Gluster setup on the cluster and I have no idea on how to recreate the LVM partitions, gluster bricks, and then add it to the cluster in order to start the healing process. Any help on how to proceed with this scenario will be very welcome. Ramon

2 1

oVirt 4.3.6 and Security Measures
by scroodj＠gmail.com 10 Mar '21

10 Mar '21

Hello team, Due to security policy in the our customer`s company there is need to implement some changes into machines in their oVirt cluster (Standalone Engine + 2 KVM Host). 1. The home drives of user sanlock (/var/run/sanlock) and gluster (/run/gluster) have permission of 775. We would like to have them at least 755 if not stricter. Is that possible? 2. NFS mount of storage has ‘nodev’ and ‘nosuid’ disabled. Is it safe to use those options for NFS Storage doamin? 3. Usually bridged routing is not allowed on managed servers. Security scan asks us to set the following four parameters to 0 Network Parameter "net.ipv4.conf.all.send_redirects" = 1 (expected: 0) Network Parameter "net.ipv4.conf.all.secure_redirects" = 1 (expected: 0) Network Parameter "net.ipv6.conf.all.accept_redirects" = 1 (expected: 0) Network Parameter "net.ipv4.conf.all.accept_redirects" = 1 (expected: 0) Would changing them interfere with ovirtmgmt network? Those are valid for all three machines in the cluster. On the engine though there is httpd installed now and we have some findings there too: 1. There are modules installed that are on a blacklist. Can they be removed? The modules are: mod_dav_lock mod_userdir mod_include mod_dav_fs mod_autoindex mod_dav mod_info 2. HTTP traces should be blocked so we would set “TraceEnable” to off in virtual host config. If HTTP traces are needed we would have to limit the verbs that are allowed. 3. Apache version information should be turned off to not inform potential attackers of which web server is running. Is that a problem for oVirt? 4. TLSv1.0 and TLSv1.1 are enabled but should be turned off. 5. HSTS should be turned on but is not yet. 6. Can we use X-Frame-Options header to append X-Frame-Options DENY (or SAEMORIGIN or at least ALLOW-FROM)? 7. Can we implement the X-Content-Type-Options HTTP header with “nosniff”? 8. Can we implement the X-XSS-Protection header with “1; mode=block”? I know, this is quite a bit. But maybe you know the answers. BR Aleksandr

3 3

Set host CPU type to kvm64 for a single VM
by Andrei Verovski 10 Mar '21

10 Mar '21

Hi ! Is it possible to set host CPU type to kvm64 for a single VM ? Thanks. Andrei

1 0

engine - gluster volume import
by penguin pages 10 Mar '21

10 Mar '21

I keep going through cycles to get a HCI cluster to deploy. Gluster is working fine. Standard from HCI wizard: Brick thorst.penguinpages.local:/gluster_br icks/vmstore/vmstore 49154 0 Y 40968 Brick odinst.penguinpages.local:/gluster_br icks/vmstore/vmstore 49157 0 Y 3771 Brick medusast.penguinpages.local:/gluster_ bricks/vmstore/vmstore 49154 0 Y 5138 Self-heal Daemon on localhost N/A N/A Y 41172 Self-heal Daemon on medusast.penguinpages.l ocal N/A N/A Y 5150 Self-heal Daemon on odinst.penguinpages.loc al N/A N/A Y 3142 What I think happens when the engine is installed "on existing disk" is that it does not install components for gluster needed to present those to be deployed upon by gluster. What is this service / install / means to get the engine to "add" gluster volumes? Ex: When I click -> Storage -> (no volumes listed) -> New volume -> <empty>.... all drops downs are blank and cannot be populated.

2 2

Commvault
by Colin Coe 10 Mar '21

10 Mar '21

Hi all My workplace is considering replacing Arcserve (which has been terrible) with Commvault. Our virtualisation is a mix of about 95% RHV and 5% HyperV. We have a handful of physical servers, a 50/50 mix of Windows and RHEL. We run dual hot data centers, and want to backup locally (disk to disk) then replicate to the other data center. Cloud is off the table. Anyone have any experience with Commvault? War stories? Happy to be pointed at other alternatives. Thanks

4 6

Setup Hosts Network Disabled
by Andrei Verovski 10 Mar '21

10 Mar '21

Hi ! I run into a problem which looks like a software bug. Network -> Networks -> My_Net_Name -> Hosts Setup Hosts Network button is disabled (greyed out). I deleted this network, created again, restarted hosted engine - no changes. Is it possible to fix this for example from command line ? Thanks in advance.

2 2

ERROR: Installing oVirt Node & Hosted-Engine on one physical server
by ivanpashchuk＠ipoft.com 10 Mar '21

10 Mar '21

oVirt node 4.4.4 Question 1: In General, can i install oVirt Node and hosted-engine on one physical server? Question 2: What domain names should I specify in the DNS server and which ones IP should I reserve? Question 3: "Failed to connect to the host via ssh: ssh: connect to host ovirt-engine-01.local port 22: No route to host" - Who and where is trying to establish a connection via SSH? Can someone reply to questions 1-3 above? Any help will be much appreciated. Thanks for the help, Ivan. Logs: ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"attempts": 180, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.565239", "end": "2021-03-03 13:36:54.775239", "rc": 0, "start": "2021-03-03 13:36:54.210000", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"host-id\": 1, \"host-ts\": 16377, \"score\": 3400, \"engine-status\": {\"vm\": \"up\", \"health\": \"bad\", \"detail\": \"Up\", \"reason\": \"failed liveliness check\"}, \"hostname\": \"mng-ovirt-engine-01.local\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"2b3ee5d1\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 16377, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=16377 (Wed Mar 3 13:36:48 2021)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=16377 (Wed Mar 3 13:36:48 2021)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStarting\\nstopped= False\\n\", \"live-data\": true}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"host-id\": 1, \"host-ts\": 16377, \"score\": 3400, \"engine-status\": {\"vm\": \"up\", \"health\": \"bad\", \"detail\": \"Up\", \"reason\": \"failed liveliness check\"}, \"hostname\": \"mng-ovirt-engine-01.local\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"2b3ee5d1\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 16377, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=16377 (Wed Mar 3 13:36:48 2021)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=16377 (Wed Mar 3 13:36:48 2021)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStarting\\nstopped=False\\n\", \"live-data\": true}, \"global_maintenance\": false}"]} ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM IP address is while the engine's he_fqdn ovirt-engine-01.local resolves to 10.0.2.250. If you are using DHCP, check your DHCP reservation configuration"} ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host ovirt-engine-01.local port 22: No route to host", "skip_reason": "Host localhost is unreachable", "unreachable": true}

2 2

Strange Glitches with VM and Network (oVirt 4.4.4.7-1.el8)
by Andrei Verovski 10 Mar '21

10 Mar '21

Hi, I’m running oVirt 4.4.4.7-1.el8 and need to connect one of the VMs straight to the ISP link via Ethernet cable. oVirt already have 2 networks (ovirt mgmt local + DMZ). Created new network provider and assigned to it available physical interface of HP ProLiant, connected via cable to ISP switch with public IP. Options of this network: (VM Network = on, Port Isolation = off, NIC Type = VirtIO, the rest are defaults). VM is Debian 10. Link works but with strange artefacts. If VM left being idle for a while, it cant be connected or pinged from outside, until I initiate pings from VM itself. I have only 2 IPs from this ISP so I’m sure there are no IP address conflicts. Another port and public IP go to our VyOS router handling internal and DMZ zone. How to fix this problem ? Thanks in advance. Andrei

2 3

which functionality replaces vdsm-hook-vmdisk in ovirt 4.4 / CentOS 8 (and is it really deprecated?)
by Andrejs Baulins 09 Mar '21

09 Mar '21

Hi there! I'm a newbie, so don't blame me fast :) I have few questions related to Hypervisor's local disk passthrough to VM strategies Preamble: we found vmdisk hook most comfortable way to use as a "gate to local storage functionality" in shared clusters. Let's assume, You have A. 2 or 4 TB large SSD, which You need to split on different usages: some diskspace provide for SQL / ETCd / Mongod etc. It must be fast, so no network in between. You have only one physical disk, so you can't passthrough whole of it to single VM. So you make LVM, and passthrough LVMs using vmdisk - this is how we did on some setups. B. 2 or 4 TB HDD for gluster bricks. At the end, You have few VMs, which "dies" together with Hypervisor, but most of VMs can live migration. In together with L7 application level fail-over, it can lead to non-critical level services level cluster, which is not so hard to throw in maintenance Yes, it does not work at real device speeds: simple dd operation with unpartitioned block device performs 3-4 times slower in VM, than on hypervisor. But, ability to mix workloads (migratable VMS with non migratable) within small cluster of hypervisors is more important, at some setups. So, there are questions coming... 1. As far as I understand, there is possibility to replace vmdisk-hook vith cmdline. But I didn't found a way, how to get it working. I always ended with "Could not open '/dev/***': No such file or directory." which I could not avoid anyhow. May be, someone can point me to an article or other thread, where there are clues, how to do that? 2. what's the story behind vmdisk hook? As far as I understand from https://github.com/oVirt/vdsm/blob/master/vdsm.spec.in its deprecated? Why? It is replaced with some functionality? Why did VDSM's hook repo been deleted from github (in order to avoid confuse?)? 3. Is it possible to achieve with any hook or some other instrument near hypervisor's performance of local disk? We are ready for inconveniences like custom setup handling / patches and so on Thanks in advance! It worked in 4.3.10 / CentOS 7, but the

1 0

HCI upgrade path from single to three-nodes setup
by s.buelow＠messaufficio.com 09 Mar '21

09 Mar '21

Hello I'm planning an HCI configuration on a single server. In my wishlist there's also the opportunity in the future to upgrade the cluster from single to fully fledged three node setup. Is there any supported and tested migration path available ? Thanks in advance Best regards

2 1

oVirt 4.3.9 readonly filesystem on multi path link failure
by alexio79＠gmail.com 09 Mar '21

09 Mar '21

Hello everyone, I have a cluster with 4 oVirt Node and a self hosted engine, fiber channel storage for storage volumes and for boot in san of oVirt Node itself (nodes doesn't have physical disks but boots from SAN). We made a software update of SAN storage and one of the multi path links went down for some seconds before coming up again. No issues with VMs but oVirt Node marked as readonly a couple of mount points: /dev/mapper/onn-var on /var type ext4 (ro,relatime,seclabel,discard,stripe=16,data=ordered) /dev/mapper/onn-var_log on /var/log type ext4 (ro,relatime,seclabel,discard,stripe=16,data=ordered) Every other mount point was r/w but not /var... We had to reboot every single node to restore r/w mounts. These are the last logs written before read only: Mar 4 11:15:59 ovirt-node1 kernel: sd 1:0:0:4: [sds] FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK Mar 4 11:15:59 ovirt-node1 kernel: sd 1:0:0:4: [sds] CDB: Read(16) 88 00 00 00 00 01 64 5f 9a 18 00 00 00 20 00 00 Mar 4 11:15:59 ovirt-node1 kernel: blk_update_request: I/O error, dev sds, sector 5978954264 Mar 4 11:15:59 ovirt-node1 kernel: device-mapper: multipath: Failing path 65:32. Mar 4 11:15:59 ovirt-node1 multipathd: sds: mark as failed Mar 4 11:15:59 ovirt-node1 multipathd: 36001738c7c8069bb00000000000135eb: remaining active paths: 3 Mar 4 11:16:11 ovirt-node1 sanlock[10535]: 2021-03-04 11:16:11 28402637 [13668]: s4 delta_renew read timeout 10 sec offset 0 /dev/58b54b5f-993b-4710-b107-7744018d22b And this is multipath -ll output: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX dm-2 IBM ,XXXXXX size=237G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 0:0:0:2 sdc 8:32 active ready running |- 1:0:0:2 sdg 8:96 active ready running |- 1:0:1:2 sdn 8:208 active ready running `- 0:0:1:2 sdr 65:16 active ready running Any idea on what happed and how to prevent it? Thank you in advice for you help.

1 0

Public IP routing question
by David White 09 Mar '21

09 Mar '21

If I have a private network (10.1.0.0/24) that is being used by the cluster for intra-host communication & replication, how do I get a block of public IP addresses routed to the virtual cluster? For example, let's say I have a public /28, and let's use 1.1.1.0/28 for example purposes. I'll assign 1.1.1.1 to the router. How can I then route 1.1.1.2 - 1.1.1.16 down to the virtualized oVirt cluster? Do I need to assign a public IP address to a 2nd physical NIC on each host, and put that network onto a totally different physical switch? Or should I instead setup default routes on the 10.1.0.0/24 network? I also wanted to follow up on my question below to see if anyone had any thoughts on how things would function when a portion of the network is lost. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Thursday, March 4, 2021 4:53 AM, David White <dmwhite823(a)protonmail.com> wrote: > I tested oVirt (4.3? I can't remember) last fall on a single host (hyperconverged). > > Now, I'm getting ready to deploy to a 3 physical node (possibly 4) hyperconverged cluster, and I guess I'll go ahead and go with 4.4. > Although Red Hat's recent shift of CentOS 8 to the Stream model, as well as the announcement that RHV is going away makes me nervous. I really don't see any other virtualization software doing quite the same stuff as oVirt at the moment. > > One of my questions is around the back end out-of-band network for data replication. > What happens if all 3 servers are healthy and the normal network is fine for serving traffic to the VM consumers, but the switching network for data replication goes down? Is it possible to configure oVirt to "fail over" to the front-end network? > I'm also wondering if its possible to do away with a switch all together, and just link the physical hosts together directly (like a cross-over cable) for the data replication. > > I'm also wondering what would happen in the following scenario: > > - All 3 servers are healthy > - The out-of-band data replication network is healthy > - 1 or 2 of the servers suddenly lost network connectivity on the front-end network > > What then? Would everything just keep working, and network traffic be forced to go out the healthy interface(s) on the remaining hosts? > > Sent with ProtonMail Secure Email.

2 2

Re: deploy oVirt 4.4 errors
by Andrew Lukoshko 09 Mar '21

09 Mar '21

I’ve submitted patch to oVirt CR: https://gerrit.ovirt.org/c/ovirt-engine/+/113817

1 0

deploy oVirt 4.4 errors
by grig.4n＠gmail.com 09 Mar '21

09 Mar '21

grep ERROR /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20210227232700-gil2fj.log ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9786 2021-02-28 00:00:12,059+0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ovirtsdk4.ConnectionError: Error while sending HTTP request: (7, 'Failed to connect to ovirt4-adm.domain.local port 443: No route to host') 2021-02-28 00:00:12,160+0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": false, "msg": "Error while sending HTTP request: (7, 'Failed to connect to ovirt4-adm.domain.local port 443: No route to host')"} 2021-02-28 00:00:58,055+0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} 2021-02-28 00:00:58,759+0600 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': Failed executing ansible-playbook 2021-02-28 00:01:05,984+0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host ovirt4-adm.domain.local. port 22: No route to host", "skip_reason": "Host localhost is unreachable", "unreachable": true} 2021-02-28 00:01:22,146+0600 ERROR otopi.plugins.gr_he_common.core.misc misc._terminate:167 Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.

3 17

HCI Fresh Deploy - Cannot upgrade 4.4. to 4.5 or add more hosts
by penguin pages 08 Mar '21

08 Mar '21

I reinstalled OS all nodes CentoOS 8 streams. installed cockpit engine and ran through HCI deploy wizard with gluster. this deployed 4.4 version of engine.. then said 4.5 version available. I try to add hosts and get error "Error while executing action: Server thor.penguinpages.local is already part of another cluster." (or other node). I ssh into ovirt engine host and attempt to ssh from that host to three nodes of HCI cluster and it works fine. I added ssh keys for passwordless login but no change I change cluster then datacenter to 4.5. Then attempt to run upgrade but get same error. Seems like chicken / egg issue. I can't get nodes. I can't upgrade 4.4. to 4.5 because I have no nodes to allow maintenance mode. And engine shows "no updates" needed [root@ovirte01 ~]# yum update Last metadata expiration check: 1:39:16 ago on Mon 08 Mar 2021 08:36:40 AM EST. Dependencies resolved. Nothing to do. Complete! Suggestions?

1 2

[security-updates] oVirt node
by Thiago Linhares 08 Mar '21

08 Mar '21

Hello there, I wonder whats the right approach to get security updates for ovirt nodes? (installed using ovirt node iso image) Eg.: 'sudo' package has a know vulnerability until version sudo-1.8.23-9.el7.x86_64. When trying to update this package on a ovirt node, it would not. Checking versionlock, I confirmed it is listed there: # grep sudo /etc/yum/pluginconf.d/versionlock.list 0:libsss_sudo-1.16.4-37.el7_8.3.x86_64 0:sudo-1.8.23-9.el7.x86_64 Regards,

2 1

CVE-2021-3156 && ovirt-node-ng 4.3 && 4.4 (sudo)
by Renaud RAKOTOMALALA 08 Mar '21

08 Mar '21

Hello everyone, I operate several oVirt clusters including pre-productions using ovirt-node-ng images. For our traditional clusters we manage the incident in a unitary way with a dedicated rpm, however for ovirt-node-ng I am not yet up to date with critical package updates process. Do you have any advice or tips? Nice day, Renaud

3 2

Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Marko Vrgotic 08 Mar '21

08 Mar '21

The broker log, these lines are pretty much repeating: MainThread::WARNING::2021-03-03 09:19:12,086::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None' MainThread::INFO::2021-03-03 09:19:12,829::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started MainThread::INFO::2021-03-03 09:19:12,829::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/sub monitors MainThread::INFO::2021-03-03 09:19:12,829::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2021-03-03 09:19:12,834::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network MainThread::INFO::2021-03-03 09:19:12,836::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain MainThread::INFO::2021-03-03 09:19:12,836::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::WARNING::2021-03-03 09:19:12,836::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None' MainThread::INFO::2021-03-03 09:19:13,574::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started MainThread::INFO::2021-03-03 09:19:13,575::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors MainThread::INFO::2021-03-03 09:19:13,575::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2021-03-03 09:19:13,577::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2021-03-03 09:19:13,578::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 8 March 2021 at 15:34 To: Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue Hi Didi, Please find the attached logs from Host and Engine. Host ovirt-sj-02 HE Undeploy 2021-03-08 14:15:52 till 2021-03-08 14:18:24 Host ovirt-sj-02 HE Deploy 2021-03-08 14:20:51 till 2021-03-08 14:23:22 I do see errors in the agent and broker and vdsm, but I do not see why it happened. Thank you for helping, let me know if any additional files are needed. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Monday, 8 March 2021 at 09:25 To: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** Hi, On Mon, Mar 8, 2021 at 10:13 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote: > > I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before. > > No errors, found int the deployment, but it seems half done, based on messages I sent in previous email. Please check/share all relevant logs. Thanks. Can be all of /var/log from engine and hosts, and at least: /var/log/ovirt-engine/engine.log /var/log/vdsm/* /var/log/ovirt-hosted-engine-ha/* Best regards, -- Didi

1 0

Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Marko Vrgotic 08 Mar '21

08 Mar '21

Hi Didi, Please find the attached logs from Host and Engine. Host ovirt-sj-02 HE Undeploy 2021-03-08 14:15:52 till 2021-03-08 14:18:24 Host ovirt-sj-02 HE Deploy 2021-03-08 14:20:51 till 2021-03-08 14:23:22 I do see errors in the agent and broker and vdsm, but I do not see why it happened. Thank you for helping, let me know if any additional files are needed. I will send the broker.log separately ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 8 March 2021 at 15:29 To: Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue Hi Didi, Please find the attached logs from Host and Engine. Host ovirt-sj-02 HE Undeploy 2021-03-08 14:15:52 till 2021-03-08 14:18:24 Host ovirt-sj-02 HE Deploy 2021-03-08 14:20:51 till 2021-03-08 14:23:22 I do see errors in the agent and broker and vdsm, but I do not see why it happened. Thank you for helping, let me know if any additional files are needed. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Monday, 8 March 2021 at 09:25 To: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** Hi, On Mon, Mar 8, 2021 at 10:13 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote: > > I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before. > > No errors, found int the deployment, but it seems half done, based on messages I sent in previous email. Please check/share all relevant logs. Thanks. Can be all of /var/log from engine and hosts, and at least: /var/log/ovirt-engine/engine.log /var/log/vdsm/* /var/log/ovirt-hosted-engine-ha/* Best regards, -- Didi

1 0

sanlock error
by kim.kargaard＠noroff.no 08 Mar '21

08 Mar '21

Hi, We are running ovirt 4.3 and whenever we create a new VM, we get the error: HSMGetAllTasksStatusesVDS failed: Error creating a new volume: (u"Volume creation 77be733a-a38a-4125-a3ae-beea960d9e28 failed: (28, 'Sanlock resource write failure', 'No space left on device')",) 3/8/2111:56:57 AM I have checked the engine logs and see the following: 2021-03-08 12:09:18,306+01 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [] BaseAsyncTask::logEndTaskFailure: Task 'd4aa81a1-9adc-45c9-be46-05bc63862f4e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Volume does not exist: (u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201', -- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Volume does not exist: (u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201' Not sure how to solve this issue. I have tried restarting the engine, but that did not solve it. Any tips? Thanks.

3 4

Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Yedidyah Bar David 08 Mar '21

08 Mar '21

Hi, On Mon, Mar 8, 2021 at 10:13 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote: > > I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before. > > No errors, found int the deployment, but it seems half done, based on messages I sent in previous email. Please check/share all relevant logs. Thanks. Can be all of /var/log from engine and hosts, and at least: /var/log/ovirt-engine/engine.log /var/log/vdsm/* /var/log/ovirt-hosted-engine-ha/* Best regards, -- Didi

1 0

Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Marko Vrgotic 08 Mar '21

08 Mar '21

I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before. No errors, found int the deployment, but it seems half done, based on messages I sent in previous email. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Vrgotic, Marko <M.Vrgotic(a)activevideo.com> Date: Wednesday, 3 March 2021 at 13:12 To: users(a)ovirt.org <users(a)ovirt.org> Subject: Upgrade from 4.3.5 to 4.3.10 HE Host issue Hi oVirt, Yesterday I upgraded one of my oVIrt platform from 4.3.5 to 4.3.10. Platform has 10Hosts with 3Hosts as Hosts for HA. Before upgrade, all three Hosts were acknowledged Hosts for HE (little happy crown). Pre upgrade: Host1 – HE available host Host2 – HE current host Host3 – HE available host Upgraded first HE to 4.3.10, than started with HE Hosts – post upgrade of Host2, the crown disappeared and did not show up. This happens sometimes, and I usually reinstall the host, than execute HE UNDEPLOY and DEPLOY, which fixes the state. However, this time, after such actions, the server is still not an acknowledged HE host – no crown. However, if I try to remove it, oVIrt says that looks like HE is deploy on it and need to be removed, but command “hosted-engine –vm-status” says you must deploy first. I removed the Host, reinstalled the OS on it, deployed it again as new, synced all networks, than deploy once more with HE Deploy, still same issue. Any suggestions? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

4.4.4 Image Copying / Template Create Fails - No Such File
by Alex McWhirter 08 Mar '21

08 Mar '21

I've been wrestling with this all night, digging through various bits of VDSM code trying to figure why and how this is happening. I need to make some templates, but i simply can't. VDSM <host> command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed: ("Command ['/usr/bin/qemu-img', 'convert', '-p', '-t', 'none', '-T', 'none', '-f', 'raw', '-O', 'raw', '/rhev/data-center/mnt/glusterSD/<gluster>:_Temp/45740f16-b3c9-4bb5-ba5f-3e64657fb663/images/6f87b073-c4ec-42f2-87da-d1cb6a08a150/f2dfb779-b49c-4ec9-86cd-741f3fe5b781', '/rhev/data-center/mnt/glusterSD/<gluster>:_Temp/45740f16-b3c9-4bb5-ba5f-3e64657fb663/images/84e56da6-8c26-4518-80a4-20bc395214db/039c5ada-ad6d-45c0-8393-bd4db0bbc366'] failed with rc=1 out=b'' err=bytearray(b'qemu-img: error while writing at byte 738197504: No such file or directory\\n')",) abortedcode=261 3/7/21 5:44:44 AM Following the VDSM logs, i can see the new image gets created, permissions set, etc... but as soon qemu-img starts, it fails like this. I updated all hosts and the engine, rebooted the entire stack, to no avail. So i detached the storage domain, and wiped every host and fresh installed both engine and all nodes, imported the storage domain, and still no dice. Storage domain is gluster volume, single node, created in ovirt. It happens when i make a template, copy an image, or make a new vm from a template. I can still create new vms from blank, and upload images via the web ui. Watching the gluster share, i can see the image being created, but its deleted at some point. I appears to not be being deleted by the template / copying process, as immediately after the above error, i get this one. VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: 'image=4f359545-01a8-439b-832b-18c26194b066, domain=b4507449-ac40-4e35-be66-56441bb696ac' 3/7/21 5:44:44 AM I thought maybe garbage collection, but don't see any indication of that in the logs. Any ideas? I redacted host names from the log output.

3 5

Which Operating System to choose
by sinan＠turka.nl 07 Mar '21

07 Mar '21

I want to set up a small cluster consisting of 2 Hypervisor nodes (self-hosted engine) with shared storage (SAN, ISCSI). According to https://www.ovirt.org/download/ the following Operating Systems are supported with the latest oVirt version (4.4.4): - Red Hat Enterprise Linux 8.3 - CentOS Linux 8.3 - CentOS Stream (Tech Preview) My doubts are: - RHEL 8.3 is free to use up tp 16 hosts, but until when? Red Hat can decide at anytime to change their subscription program. - CentOS Linux is no more, support will stop at the end of 2021. - CentOS Stream is in Tech Preview. I don't like running production environments on "tech preview / beta" versions. I would say CentOS 7, it will receive maintenance updates up to 2024, and switch to RHEL in 2024 if the free-to-use program is still active. Unfortunately, CentOS 7 seems not to be supported. On what Operating System are you running oVirt 4.4.4 and why? Any feedback and advice is welcome. Thanks!

2 1

Cold Move VM Storage failing
by dmmailing010203＠gmail.com 07 Mar '21

07 Mar '21

Morning oVirt. I've rebuilt my oVirt environment and I'm currently on 4.4.5.8-1.el8. I have storage presented via NFS. I'm unable to move / copy VM disks when powered off. However I can Live migrate the disks when powered on. I have three storage domains presented and this is repeated across all three no matter where the storage is located. The disks are thin provisioned. I've tried deactivating and detaching the disk and it's the same NullPointerException error. 2021-03-06 15:13:17,010Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-84) [af63add0-0ef3-4049-8461-b564abedba02] Command 'org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand' failed: null 2021-03-06 15:13:17,010Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-84) [af63add0-0ef3-4049-8461-b564abedba02] Exception: java.lang.NullPointerException Engine log with stack trace is at https://textuploader.com/1820f Vdsm log is at https://textuploader.com/1820z I'm just checking I've not missed something obvious before raising a bug if neccessary

2 2

Gluster version upgrades
by Sketch 07 Mar '21

07 Mar '21

Is the gluster version on an oVirt host tied to the oVirt version, or would it be safe to upgrade to newer versions of gluster? I have noticed gluster is often updated to new major versions on oVirt point release upgrades. We have some compute+storage hosts on 4.3.6 which can't be upgraded easily at the moment, but we are having some gluster issues that appear to be due to bugs that I wonder if upgrading might help. Would an in-place upgrade of gluster be a bad idea without also updating oVirt?

2 1

VM ovirt-win10-01 is down with error. Exit message: Unable to get volume size for domain 5cee5082-074e-4a8b-a750-0e8fb0aff63c volume 05063eec-ca54-42c6-92d6-0a6775545eba.
by wangyu13476969128＠126.com 06 Mar '21

06 Mar '21

The version of ovirt-manager is 4.3.6.7。 I have two ovirt-nodes. After I restart one ovirt-node, all the Linux virtual machines on the ovirt-node can startup. But all the Windows virtual machines on the ovirt-node can't startup。 The title show the error: VM ovirt-win10-01 is down with error. Exit message: Unable to get volume size for domain 5cee5082-074e-4a8b-a750-0e8fb0aff63c volume 05063eec-ca54-42c6-92d6-0a6775545eba. How can I solve this problem ?

1 0

Removed FC LUN causes multipath to spam syslog
by Juhani Rautiainen 06 Mar '21

06 Mar '21

Hi! We are running ovirt 4.3.10 and we are migrating from 3PAR to Dell SC. I've managed to transfer disks from one LUN away and removed it (put it in maintenanced, detached it and removed it). Now multipath seems to spam the logs for missing disks. How can I stop this? Mar 5 09:58:58 ovirt01 multipathd: 360002ac00000000000000272000057b9: sdr - tur checker reports path is down Mar 5 09:58:59 ovirt01 multipathd: 360002ac00000000000000272000057b9: sdh - tur checker reports path is down And so on. Any idea how I can quiet this? And why didn't oVirt do this automatically? Thanks, Juhani

3 7

Re: EL7 Host support in 4.4.x
by Nir Soffer 04 Mar '21

04 Mar '21

On Thu, Mar 4, 2021 at 6:06 PM Vrgotic, Marko <M.Vrgotic(a)activevideo.com> wrote: > Hi oVirt, > > > > After reviewing the https://access.redhat.com/articles/973163 > > I wonder is it possible to have a Host running EL7, in oVirt 4.4 to > support ability to still run OSs like Win 2008R2, or is it a hard NO? > Theoretically vdsm can run on EL7 if you install python3, but without libvirt and qemu versions required in 4.4 it will not be very useful. If you need EL7, use ovirt 4.3. Nir

2 1

Create update and security plan
by Gilson Schmidt 04 Mar '21

04 Mar '21

Hi We have a large hypervisors park, and now we are starting to adopt oVirt. And I need an update and security plan for oVirt Nodes. I need to know: - What is the update cycle for oVirt? - How large companies treat the security of oVirt Node. What is the frequency of updates; Is an additional service enabled / installed on oVirt Node? Is SSH enabled maintained? What is the sysadmin access format? Currently on other hypervisors (Xen) using Vault, or we disable SSH on ESXi. How do large companies configure oVirt to comply with the needs passed on by the Security Team? Tks very much.

2 1

Difference between edit cluster and upgrade?
by Gianluca Cecchi 04 Mar '21

04 Mar '21

Hello, supposing I upgrade my env from 4.3 to 4.4. Then I can update my clusters' compatibility from 4.3 to 4.5. I can do it in two ways: 1) select cluster line, "edit" button and change "Compatibility Version" from 4.3 to 4.5 2) select cluster line, "upgrade" button correct? What is the difference? Thanks, Gianluca

1 0

[ANN] oVirt 4.4.5 Eighth Release Candidate is now available for testing
by Lev Veyde 04 Mar '21

04 Mar '21

oVirt 4.4.5 Eighth Release Candidate is now available for testing The oVirt Project is pleased to announce the availability of oVirt 4.4.5 Eighth Release Candidate for testing, as of March 4th, 2021. This update is the fifth in a series of stabilization updates to the 4.4 series. How to prevent hosts entering emergency mode after upgrade from oVirt 4.4.1 Note: Upgrading from 4.4.2 GA or later should not require re-doing these steps, if already performed while upgrading from 4.4.1 to 4.4.2 GA. These are only required to be done once. Due to Bug 1837864 <https://bugzilla.redhat.com/show_bug.cgi?id=1837864> - Host enter emergency mode after upgrading to latest build If you have your root file system on a multipath device on your hosts you should be aware that after upgrading from 4.4.1 to 4.4.5 you may get your host entering emergency mode. In order to prevent this be sure to upgrade oVirt Engine first, then on your hosts: 1. Remove the current lvm filter while still on 4.4.1, or in emergency mode (if rebooted). 2. Reboot. 3. Upgrade to 4.4.5 (redeploy in case of already being on 4.4.5). 4. Run vdsm-tool config-lvm-filter to confirm there is a new filter in place. 5. Only if not using oVirt Node: - run "dracut --force --add multipath” to rebuild initramfs with the correct filter configuration 6. Reboot. Documentation - If you want to try oVirt as quickly as possible, follow the instructions on the Download <https://ovirt.org/download/> page. - For complete installation, administration, and usage instructions, see the oVirt Documentation <https://ovirt.org/documentation/>. - For upgrading from a previous version, see the oVirt Upgrade Guide <https://ovirt.org/documentation/upgrade_guide/>. - For a general overview of oVirt, see About oVirt <https://ovirt.org/community/about.html>. Important notes before you try it Please note this is a pre-release build. The oVirt Project makes no guarantees as to its suitability or usefulness. This pre-release must not be used in production. Installation instructions For installation instructions and additional information please refer to: https://ovirt.org/documentation/ This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 8.3 or newer * CentOS Linux (or similar) 8.3 or newer This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 8.3 or newer * CentOS Linux (or similar) 8.3 or newer * oVirt Node 4.4 based on CentOS Linux 8.3 (available for x86_64 only) See the release notes [1] for installation instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available for CentOS Linux 8 - oVirt Node NG is already available for CentOS Linux 8 - We found a few issues while testing on CentOS Stream so we are still basing oVirt 4.4.5 Node and Appliance on CentOS Linux. Additional Resources: * Read more about the oVirt 4.4.5 release highlights: http://www.ovirt.org/release/4.4.5/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.4.5/ [2] http://resources.ovirt.org/pub/ovirt-4.4-pre/iso/ -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

1 0

EL7 Host support in 4.4.x
by Vrgotic, Marko 04 Mar '21

04 Mar '21

Hi oVirt, After reviewing the https://access.redhat.com/articles/973163 I wonder is it possible to have a Host running EL7, in oVirt 4.4 to support ability to still run OSs like Win 2008R2, or is it a hard NO? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

Manager login error
by Moses Mbugua 04 Mar '21

04 Mar '21

have tried to resolve this error " Keystore was tampered with, or password was incorrect" but am not able. This has rendered my virtualization manager unusable. This means I can access one of my hosts which has vms l down. any one who has ever faced similar issues? i rarely access it thus mostly an update issue since i can't remember changing anything .redhat virtualization version 4.2.8.9-0.1.el7ev. able to help? ping me we discuss

3 5

What happens in loss of network scenarios?
by David White 04 Mar '21

04 Mar '21

I tested oVirt (4.3? I can't remember) last fall on a single host (hyperconverged). Now, I'm getting ready to deploy to a 3 physical node (possibly 4) hyperconverged cluster, and I guess I'll go ahead and go with 4.4. Although Red Hat's recent shift of CentOS 8 to the Stream model, as well as the announcement that RHV is going away makes me nervous. I really don't see any other virtualization software doing quite the same stuff as oVirt at the moment. One of my questions is around the back end out-of-band network for data replication. What happens if all 3 servers are healthy and the normal network is fine for serving traffic to the VM consumers, but the switching network for data replication goes down? Is it possible to configure oVirt to "fail over" to the front-end network?I'm also wondering if its possible to do away with a switch all together, and just link the physical hosts together directly (like a cross-over cable) for the data replication. I'm also wondering what would happen in the following scenario: - All 3 servers are healthy - The out-of-band data replication network is healthy - 1 or 2 of the servers suddenly lost network connectivity on the front-end network What then? Would everything just keep working, and network traffic be forced to go out the healthy interface(s) on the remaining hosts? Sent with ProtonMail Secure Email.

1 0

Cloud-init confusion
by Shantur Rathore 04 Mar '21

04 Mar '21

Hi Users, I am new to oVirt and having some confusion with cloud-init. I have a VM which supports Cloud Init and when I enable Cloud Init under Initial boot settings of VM, the cloud-init config disk isn't attached to the VM. If I use Run-Once and enable Cloud-Init the config disk is attached to the VM and works as expected. Is cloud-init only supposed to work with Run-Once or am I missing something? Thanks in advance. Shantur

2 2

Template backing disks on different storage
by Shantur Rathore 04 Mar '21

04 Mar '21

Hi Users, I am evaluating oVirt coming from a VMware background. I have a test cluster with 6 machines running oVirt Node providing compute + gluster services. There are 2 storage domains 1. Gluster backed 2. iSCSI SAN I use iSCSI SAN for the templates (backing disks) and would like to use Gluster storage domain for template instances. In VMware I could create a linked clone on one storage linked to a template on another. Is this possible with oVirt? via GUI or API? Thanks in advance, Shantur

1 0

node IPs can't be pingued on a fresh 4.4.4 installation
by Nathanaël Blanchet 04 Mar '21

04 Mar '21

Hello On a fresh installation of a 4.4.4 node, I configured network so as to get dhcp delivered IP for gluster and migration services. After IPs have been delivered, they can't be pingued and vms migrations fail. All work as expected once I reactivate each bridge with nmcli/nmtui... This bug is present on each new node installation. -- Nathanaël Blanchet Supervision réseau SIRE 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

2 1

Right Glusterfs Config on HA Server
by Marcel d'Heureuse 04 Mar '21

04 Mar '21

Hi Guys or Girls, I have a problem in a productive system. We use three Servers which have a Raid 5 with four Harddrives SATA 8 TB 7200 rpms. The Systems are interconnected for Glusterfs with 1 GB/s. The Network Load between all three servers are between 5 and 8 % of the network. The Glusterfs has a separate VLAN which complete isolated from other networks. Find attached a small drawing. On that server we rum as HA and there are 8 VMs installed. Most of the VMs working well but if there is high load on the system some of the VM can't write the Data in a "good" speed on the disks. The Glusterfs is configured as Replica on the Raid5 disk which is as Jbob in Glusterfs defined. If i take a look on the iostats i can see that the Harddisk have round 50 open connections some of the glusterfs and a lot of the vms. In side the VMs i can see long iowaiting times from 300 up to 600 ms. With is on case of a Posgresql database very long. Should we configure the Raid5 controller out that the controller only should 4 disks with 8 TB and Glusterfs should connect directly to the /dev/sd[b-e]? What do you think? Will SSD HDD will help on performance but the same config with Raid5 and Jbod could be the bigger problem. Thanks Marcel

2 1

4.5 compatibility cluster
by Nathanaël Blanchet 03 Mar '21

03 Mar '21

2 1

Newly uploaded disk stuck in 'Finalizing Failure' status
by n.bouwhuis＠speakup.nl 03 Mar '21

03 Mar '21

Hey there, A few days ago, I tried to create a new template using one of the Ansible playbooks I frequently use. This time, the upload of the disk failed, possibly due to an unrelated error. The disk is now stuck in Ovirt. It has the status 'Finalizing Failure' and it's locked, thus I cannot remove it. Trying to unlock the disk with the 'unlock_entity.sh' helper utility does not help. The utility says it has completed it task successfully but the disk remains in this state. I decided to check the logs, and see that `engine.log` gets spammed with these lines every 10 seconds ``` 2021-03-03 09:05:38,335Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-45) [redacted] Failed to stop image transfer session. Ticket does not exist for image '[disk uuid]' ``` How can I force Ovirt to remove this disk? I presume the upload failure itself is a different error, possibly related to our setup. Hence this is not the topic of this thread. I just want to know how I can remove these disks. Thanks, Nick

3 3

SHE upgrade from 4.3 to 4.4 doubts
by Gianluca Cecchi 03 Mar '21

03 Mar '21

Hello, I would like to get some clarifications about the SHE environment upgrade from 4.3.10 to 4.4.4. Example scenario two hosts, both configured as HE hosts. Interested both in iSCSI environment and in FC environment. Going through here: https://www.ovirt.org/documentation/upgrade_guide/#Upgrading_the_Manager_to… I have some doubts: 1) general one between step 5 and 6 in section 4.4, actually the current 4.3 engine VM is still running its OS (even if with ovirt-engine service stopped) and the VM is running on the host you are going to scratch. So I think before scratching the host and reinstalling it (step 6), it should be clarified some step like - shutdown engine VM - shutdown host (is it ok in clean way or do I have to simply power off it....?) - install from scratch ovirt-ng node 4.4.4 on host 2) general one why in external engine upgrade guide it is not specified to stop ovirt-engine service before executing the backup, while in SHE it is specified? See steps on section 2.3 here: https://www.ovirt.org/documentation/upgrade_guide/#Upgrading_from_4-3 expected different behaviour? Or any change needed in the doc? 3) SHE when iSCSI and FC It is clear that one new domain is to be provided, intended for the new engine 4.4 VM It is clear that during install of the new host in 4.4 it is safe to have shared LUNs masked / not accessible so that you don't risk to corrupt current 4.3 data It is not clear in my opinion what to do before running the step 8 of 4.4 section: hosted-engine --deploy --restore-from-file=backup.bck a) iSCSI in general the configured iSCSI sessions are activated only when the host is being activated. So for example when you put host into maintenance, they are logged off. In this stage of upgrade what is expected I execute before running the hosted-engine --deploy command? Will the command log in into iSCSI sessions during itsmexecution (because of the restore) or do I have to create / log in the connections before running it? b) FC what should I have on host before running the command? Only the current LUns (+ the new one intended for new hosted engine VM) visible, so that I see all of them as output of "multipath -l" command and then the hosted-engine --deploy command would mount the related filesystems and so on..? Possibly it would be nice to have an example output of the hosted-engine --deploy --restore-from-file=backup.bck command inside the doc page.... to see its expected workflow Thanks for reading, Gianluca

1 0

Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Vrgotic, Marko 03 Mar '21

03 Mar '21

Hi oVirt, Yesterday I upgraded one of my oVIrt platform from 4.3.5 to 4.3.10. Platform has 10Hosts with 3Hosts as Hosts for HA. Before upgrade, all three Hosts were acknowledged Hosts for HE (little happy crown). Pre upgrade: Host1 – HE available host Host2 – HE current host Host3 – HE available host Upgraded first HE to 4.3.10, than started with HE Hosts – post upgrade of Host2, the crown disappeared and did not show up. This happens sometimes, and I usually reinstall the host, than execute HE UNDEPLOY and DEPLOY, which fixes the state. However, this time, after such actions, the server is still not an acknowledged HE host – no crown. However, if I try to remove it, oVIrt says that looks like HE is deploy on it and need to be removed, but command “hosted-engine –vm-status” says you must deploy first. I removed the Host, reinstalled the OS on it, deployed it again as new, synced all networks, than deploy once more with HE Deploy, still same issue. Any suggestions? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

Multipath flapping with SAS via FCP
by Benoit Chatelain 02 Mar '21

02 Mar '21

Hi, I have some troubles with multipath. When I add SAS disk over FCP as Storage Domain via oVirt WebUI, The first link as active, but the second is stuck as failed. Volum disk is provided from Dell Compellent via FCP, and disk is transported in SAS. multipath is flapping in all hypervisor from the same domain disk: [root@isildur-adm ~]# tail -f /var/log/messages Feb 25 11:48:21 isildur-adm kernel: device-mapper: multipath: 253:3: Failing path 8:32. Feb 25 11:48:24 isildur-adm multipathd[659460]: 36000d31003d5c2000000000000000010: sdc - tur checker reports path is up Feb 25 11:48:24 isildur-adm multipathd[659460]: 8:32: reinstated Feb 25 11:48:24 isildur-adm multipathd[659460]: 36000d31003d5c2000000000000000010: remaining active paths: 2 Feb 25 11:48:24 isildur-adm kernel: device-mapper: multipath: 253:3: Reinstating path 8:32. Feb 25 11:48:24 isildur-adm kernel: sd 1:0:1:2: alua: port group f01c state S non-preferred supports toluSNA Feb 25 11:48:24 isildur-adm kernel: sd 1:0:1:2: alua: port group f01c state S non-preferred supports toluSNA Feb 25 11:48:24 isildur-adm kernel: device-mapper: multipath: 253:3: Failing path 8:32. Feb 25 11:48:25 isildur-adm multipathd[659460]: sdc: mark as failed Feb 25 11:48:25 isildur-adm multipathd[659460]: 36000d31003d5c2000000000000000010: remaining active paths: 1 --- [root@isildur-adm ~]# multipath -ll 36000d31003d5c2000000000000000010 dm-3 COMPELNT,Compellent Vol size=1.5T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=25 status=active |- 1:0:0:2 sdb 8:16 active ready running `- 1:0:1:2 sdc 8:32 failed ready running --- VDSM generate multipath.conf like this ( I have remove commented lines for read confort ) : [root@isildur-adm ~]# cat /etc/multipath.conf # VDSM REVISION 2.0 # This file is managed by vdsm. defaults { polling_interval 5 no_path_retry 16 user_friendly_names no flush_on_last_del yes fast_io_fail_tmo 5 dev_loss_tmo 30 max_fds 4096 } blacklist { protocol "(scsi:adt|scsi:sbp)" } no_path_retry 16 } Have you some idea why this link is flapping on my two hypervisor? Thanks a lot in advance. - Benoit Chatelain

4 5

How to enable nested virtualization
by miguel.garcia＠toshibagcs.com 02 Mar '21

02 Mar '21

Hi all, We are trying to run virtual box in a vm that is running over ovirt platform. I had followed the instruction to enable nested virtualization on host as described in this document https://ostechnix.com/how-to-enable-nested-virtualization-in-kvm-in-linux/ However I was not able to enable nested virtualization to the vm since nothing is listed from virsh console. Also tryed to edit vm configuration lookig up for "cpu mode" or "copy cpu configuration as host" Also from oVirt admin portal I had enabled host for nested virtualization through Compute > Host > selected Host > edit > Kernel Nested virtualization checked. But the host still returns: $ cat /sys/module/kvm_intel/parameters/nested N

4 5

kvm host becomes non operational after breaking a bonding interface
by LS CHENG 02 Mar '21

02 Mar '21

Hi I have created a couple of bonding interfaces in the administrator portal, due to some miss configuration I had to break the bond interfaces, what I observe is that after breaking the bonding interfaces the KVM host becomes non operational and it stays in such state unless I add the bonding interfaces back. Anyone have observed this behaviour? Thanks

2 6

2 Node oVirt Cluster
by justin.lamp96＠gmail.com 01 Mar '21

01 Mar '21

Hey there, I tried using oVirt for some time now and like to convert my main Proxmox Cluster (2 Nodes + RPI for Quorum) to oVirt. But I obvioulsy need three servers to do HA. Now I am asking myself: can't I just use the PI as an arbiter node for the Selfhosted-Engine on gluster? I tried running the playbook already, but it's always failing and even if the gluster deployment would go through, it would most likely fail aftwards, right? Before I waste too much energy and time into that, does anyone of you guys use a 2 node cluster? And no I can't add another node (energy + space + money)! Also I know with doing that I'm totally on my own etcetera Thanks!

3 6