- Users - oVirt List Archives

oVirt 4.3.6 and Security Measures
by scroodj＠gmail.com 10 Mar '21

10 Mar '21

Hello team, Due to security policy in the our customer`s company there is need to implement some changes into machines in their oVirt cluster (Standalone Engine + 2 KVM Host). 1. The home drives of user sanlock (/var/run/sanlock) and gluster (/run/gluster) have permission of 775. We would like to have them at least 755 if not stricter. Is that possible? 2. NFS mount of storage has ‘nodev’ and ‘nosuid’ disabled. Is it safe to use those options for NFS Storage doamin? 3. Usually bridged routing is not allowed on managed servers. Security scan asks us to set the following four parameters to 0 Network Parameter "net.ipv4.conf.all.send_redirects" = 1 (expected: 0) Network Parameter "net.ipv4.conf.all.secure_redirects" = 1 (expected: 0) Network Parameter "net.ipv6.conf.all.accept_redirects" = 1 (expected: 0) Network Parameter "net.ipv4.conf.all.accept_redirects" = 1 (expected: 0) Would changing them interfere with ovirtmgmt network? Those are valid for all three machines in the cluster. On the engine though there is httpd installed now and we have some findings there too: 1. There are modules installed that are on a blacklist. Can they be removed? The modules are: mod_dav_lock mod_userdir mod_include mod_dav_fs mod_autoindex mod_dav mod_info 2. HTTP traces should be blocked so we would set “TraceEnable” to off in virtual host config. If HTTP traces are needed we would have to limit the verbs that are allowed. 3. Apache version information should be turned off to not inform potential attackers of which web server is running. Is that a problem for oVirt? 4. TLSv1.0 and TLSv1.1 are enabled but should be turned off. 5. HSTS should be turned on but is not yet. 6. Can we use X-Frame-Options header to append X-Frame-Options DENY (or SAEMORIGIN or at least ALLOW-FROM)? 7. Can we implement the X-Content-Type-Options HTTP header with “nosniff”? 8. Can we implement the X-XSS-Protection header with “1; mode=block”? I know, this is quite a bit. But maybe you know the answers. BR Aleksandr

3 3

Set host CPU type to kvm64 for a single VM
by Andrei Verovski 10 Mar '21

10 Mar '21

Hi ! Is it possible to set host CPU type to kvm64 for a single VM ? Thanks. Andrei

1 0

engine - gluster volume import
by penguin pages 10 Mar '21

10 Mar '21

I keep going through cycles to get a HCI cluster to deploy. Gluster is working fine. Standard from HCI wizard: Brick thorst.penguinpages.local:/gluster_br icks/vmstore/vmstore 49154 0 Y 40968 Brick odinst.penguinpages.local:/gluster_br icks/vmstore/vmstore 49157 0 Y 3771 Brick medusast.penguinpages.local:/gluster_ bricks/vmstore/vmstore 49154 0 Y 5138 Self-heal Daemon on localhost N/A N/A Y 41172 Self-heal Daemon on medusast.penguinpages.l ocal N/A N/A Y 5150 Self-heal Daemon on odinst.penguinpages.loc al N/A N/A Y 3142 What I think happens when the engine is installed "on existing disk" is that it does not install components for gluster needed to present those to be deployed upon by gluster. What is this service / install / means to get the engine to "add" gluster volumes? Ex: When I click -> Storage -> (no volumes listed) -> New volume -> <empty>.... all drops downs are blank and cannot be populated.

2 2

Commvault
by Colin Coe 10 Mar '21

10 Mar '21

Hi all My workplace is considering replacing Arcserve (which has been terrible) with Commvault. Our virtualisation is a mix of about 95% RHV and 5% HyperV. We have a handful of physical servers, a 50/50 mix of Windows and RHEL. We run dual hot data centers, and want to backup locally (disk to disk) then replicate to the other data center. Cloud is off the table. Anyone have any experience with Commvault? War stories? Happy to be pointed at other alternatives. Thanks

4 6

Setup Hosts Network Disabled
by Andrei Verovski 10 Mar '21

10 Mar '21

Hi ! I run into a problem which looks like a software bug. Network -> Networks -> My_Net_Name -> Hosts Setup Hosts Network button is disabled (greyed out). I deleted this network, created again, restarted hosted engine - no changes. Is it possible to fix this for example from command line ? Thanks in advance.

2 2

ERROR: Installing oVirt Node & Hosted-Engine on one physical server
by ivanpashchuk＠ipoft.com 10 Mar '21

10 Mar '21

oVirt node 4.4.4 Question 1: In General, can i install oVirt Node and hosted-engine on one physical server? Question 2: What domain names should I specify in the DNS server and which ones IP should I reserve? Question 3: "Failed to connect to the host via ssh: ssh: connect to host ovirt-engine-01.local port 22: No route to host" - Who and where is trying to establish a connection via SSH? Can someone reply to questions 1-3 above? Any help will be much appreciated. Thanks for the help, Ivan. Logs: ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"attempts": 180, "changed": true, "cmd": ["hosted-engine", "--vm-status", "--json"], "delta": "0:00:00.565239", "end": "2021-03-03 13:36:54.775239", "rc": 0, "start": "2021-03-03 13:36:54.210000", "stderr": "", "stderr_lines": [], "stdout": "{\"1\": {\"host-id\": 1, \"host-ts\": 16377, \"score\": 3400, \"engine-status\": {\"vm\": \"up\", \"health\": \"bad\", \"detail\": \"Up\", \"reason\": \"failed liveliness check\"}, \"hostname\": \"mng-ovirt-engine-01.local\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"2b3ee5d1\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 16377, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=16377 (Wed Mar 3 13:36:48 2021)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=16377 (Wed Mar 3 13:36:48 2021)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStarting\\nstopped= False\\n\", \"live-data\": true}, \"global_maintenance\": false}", "stdout_lines": ["{\"1\": {\"host-id\": 1, \"host-ts\": 16377, \"score\": 3400, \"engine-status\": {\"vm\": \"up\", \"health\": \"bad\", \"detail\": \"Up\", \"reason\": \"failed liveliness check\"}, \"hostname\": \"mng-ovirt-engine-01.local\", \"maintenance\": false, \"stopped\": false, \"crc32\": \"2b3ee5d1\", \"conf_on_shared_storage\": true, \"local_conf_timestamp\": 16377, \"extra\": \"metadata_parse_version=1\\nmetadata_feature_version=1\\ntimestamp=16377 (Wed Mar 3 13:36:48 2021)\\nhost-id=1\\nscore=3400\\nvm_conf_refresh_time=16377 (Wed Mar 3 13:36:48 2021)\\nconf_on_shared_storage=True\\nmaintenance=False\\nstate=EngineStarting\\nstopped=False\\n\", \"live-data\": true}, \"global_maintenance\": false}"]} ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Engine VM IP address is while the engine's he_fqdn ovirt-engine-01.local resolves to 10.0.2.250. If you are using DHCP, check your DHCP reservation configuration"} ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host ovirt-engine-01.local port 22: No route to host", "skip_reason": "Host localhost is unreachable", "unreachable": true}

2 2

Strange Glitches with VM and Network (oVirt 4.4.4.7-1.el8)
by Andrei Verovski 10 Mar '21

10 Mar '21

Hi, I’m running oVirt 4.4.4.7-1.el8 and need to connect one of the VMs straight to the ISP link via Ethernet cable. oVirt already have 2 networks (ovirt mgmt local + DMZ). Created new network provider and assigned to it available physical interface of HP ProLiant, connected via cable to ISP switch with public IP. Options of this network: (VM Network = on, Port Isolation = off, NIC Type = VirtIO, the rest are defaults). VM is Debian 10. Link works but with strange artefacts. If VM left being idle for a while, it cant be connected or pinged from outside, until I initiate pings from VM itself. I have only 2 IPs from this ISP so I’m sure there are no IP address conflicts. Another port and public IP go to our VyOS router handling internal and DMZ zone. How to fix this problem ? Thanks in advance. Andrei

2 3

which functionality replaces vdsm-hook-vmdisk in ovirt 4.4 / CentOS 8 (and is it really deprecated?)
by Andrejs Baulins 09 Mar '21

09 Mar '21

Hi there! I'm a newbie, so don't blame me fast :) I have few questions related to Hypervisor's local disk passthrough to VM strategies Preamble: we found vmdisk hook most comfortable way to use as a "gate to local storage functionality" in shared clusters. Let's assume, You have A. 2 or 4 TB large SSD, which You need to split on different usages: some diskspace provide for SQL / ETCd / Mongod etc. It must be fast, so no network in between. You have only one physical disk, so you can't passthrough whole of it to single VM. So you make LVM, and passthrough LVMs using vmdisk - this is how we did on some setups. B. 2 or 4 TB HDD for gluster bricks. At the end, You have few VMs, which "dies" together with Hypervisor, but most of VMs can live migration. In together with L7 application level fail-over, it can lead to non-critical level services level cluster, which is not so hard to throw in maintenance Yes, it does not work at real device speeds: simple dd operation with unpartitioned block device performs 3-4 times slower in VM, than on hypervisor. But, ability to mix workloads (migratable VMS with non migratable) within small cluster of hypervisors is more important, at some setups. So, there are questions coming... 1. As far as I understand, there is possibility to replace vmdisk-hook vith cmdline. But I didn't found a way, how to get it working. I always ended with "Could not open '/dev/***': No such file or directory." which I could not avoid anyhow. May be, someone can point me to an article or other thread, where there are clues, how to do that? 2. what's the story behind vmdisk hook? As far as I understand from https://github.com/oVirt/vdsm/blob/master/vdsm.spec.in its deprecated? Why? It is replaced with some functionality? Why did VDSM's hook repo been deleted from github (in order to avoid confuse?)? 3. Is it possible to achieve with any hook or some other instrument near hypervisor's performance of local disk? We are ready for inconveniences like custom setup handling / patches and so on Thanks in advance! It worked in 4.3.10 / CentOS 7, but the

1 0

HCI upgrade path from single to three-nodes setup
by s.buelow＠messaufficio.com 09 Mar '21

09 Mar '21

Hello I'm planning an HCI configuration on a single server. In my wishlist there's also the opportunity in the future to upgrade the cluster from single to fully fledged three node setup. Is there any supported and tested migration path available ? Thanks in advance Best regards

2 1

oVirt 4.3.9 readonly filesystem on multi path link failure
by alexio79＠gmail.com 09 Mar '21

09 Mar '21

Hello everyone, I have a cluster with 4 oVirt Node and a self hosted engine, fiber channel storage for storage volumes and for boot in san of oVirt Node itself (nodes doesn't have physical disks but boots from SAN). We made a software update of SAN storage and one of the multi path links went down for some seconds before coming up again. No issues with VMs but oVirt Node marked as readonly a couple of mount points: /dev/mapper/onn-var on /var type ext4 (ro,relatime,seclabel,discard,stripe=16,data=ordered) /dev/mapper/onn-var_log on /var/log type ext4 (ro,relatime,seclabel,discard,stripe=16,data=ordered) Every other mount point was r/w but not /var... We had to reboot every single node to restore r/w mounts. These are the last logs written before read only: Mar 4 11:15:59 ovirt-node1 kernel: sd 1:0:0:4: [sds] FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK Mar 4 11:15:59 ovirt-node1 kernel: sd 1:0:0:4: [sds] CDB: Read(16) 88 00 00 00 00 01 64 5f 9a 18 00 00 00 20 00 00 Mar 4 11:15:59 ovirt-node1 kernel: blk_update_request: I/O error, dev sds, sector 5978954264 Mar 4 11:15:59 ovirt-node1 kernel: device-mapper: multipath: Failing path 65:32. Mar 4 11:15:59 ovirt-node1 multipathd: sds: mark as failed Mar 4 11:15:59 ovirt-node1 multipathd: 36001738c7c8069bb00000000000135eb: remaining active paths: 3 Mar 4 11:16:11 ovirt-node1 sanlock[10535]: 2021-03-04 11:16:11 28402637 [13668]: s4 delta_renew read timeout 10 sec offset 0 /dev/58b54b5f-993b-4710-b107-7744018d22b And this is multipath -ll output: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX dm-2 IBM ,XXXXXX size=237G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 0:0:0:2 sdc 8:32 active ready running |- 1:0:0:2 sdg 8:96 active ready running |- 1:0:1:2 sdn 8:208 active ready running `- 0:0:1:2 sdr 65:16 active ready running Any idea on what happed and how to prevent it? Thank you in advice for you help.

1 0

Public IP routing question
by David White 09 Mar '21

09 Mar '21

If I have a private network (10.1.0.0/24) that is being used by the cluster for intra-host communication & replication, how do I get a block of public IP addresses routed to the virtual cluster? For example, let's say I have a public /28, and let's use 1.1.1.0/28 for example purposes. I'll assign 1.1.1.1 to the router. How can I then route 1.1.1.2 - 1.1.1.16 down to the virtualized oVirt cluster? Do I need to assign a public IP address to a 2nd physical NIC on each host, and put that network onto a totally different physical switch? Or should I instead setup default routes on the 10.1.0.0/24 network? I also wanted to follow up on my question below to see if anyone had any thoughts on how things would function when a portion of the network is lost. Sent with ProtonMail Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Thursday, March 4, 2021 4:53 AM, David White <dmwhite823(a)protonmail.com> wrote: > I tested oVirt (4.3? I can't remember) last fall on a single host (hyperconverged). > > Now, I'm getting ready to deploy to a 3 physical node (possibly 4) hyperconverged cluster, and I guess I'll go ahead and go with 4.4. > Although Red Hat's recent shift of CentOS 8 to the Stream model, as well as the announcement that RHV is going away makes me nervous. I really don't see any other virtualization software doing quite the same stuff as oVirt at the moment. > > One of my questions is around the back end out-of-band network for data replication. > What happens if all 3 servers are healthy and the normal network is fine for serving traffic to the VM consumers, but the switching network for data replication goes down? Is it possible to configure oVirt to "fail over" to the front-end network? > I'm also wondering if its possible to do away with a switch all together, and just link the physical hosts together directly (like a cross-over cable) for the data replication. > > I'm also wondering what would happen in the following scenario: > > - All 3 servers are healthy > - The out-of-band data replication network is healthy > - 1 or 2 of the servers suddenly lost network connectivity on the front-end network > > What then? Would everything just keep working, and network traffic be forced to go out the healthy interface(s) on the remaining hosts? > > Sent with ProtonMail Secure Email.

2 2

Re: deploy oVirt 4.4 errors
by Andrew Lukoshko 09 Mar '21

09 Mar '21

I’ve submitted patch to oVirt CR: https://gerrit.ovirt.org/c/ovirt-engine/+/113817

1 0

deploy oVirt 4.4 errors
by grig.4n＠gmail.com 09 Mar '21

09 Mar '21

grep ERROR /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20210227232700-gil2fj.log ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9786 2021-02-28 00:00:12,059+0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ovirtsdk4.ConnectionError: Error while sending HTTP request: (7, 'Failed to connect to ovirt4-adm.domain.local port 443: No route to host') 2021-02-28 00:00:12,160+0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"attempts": 50, "changed": false, "msg": "Error while sending HTTP request: (7, 'Failed to connect to ovirt4-adm.domain.local port 443: No route to host')"} 2021-02-28 00:00:58,055+0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} 2021-02-28 00:00:58,759+0600 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Closing up': Failed executing ansible-playbook 2021-02-28 00:01:05,984+0600 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 fatal: [localhost]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host ovirt4-adm.domain.local. port 22: No route to host", "skip_reason": "Host localhost is unreachable", "unreachable": true} 2021-02-28 00:01:22,146+0600 ERROR otopi.plugins.gr_he_common.core.misc misc._terminate:167 Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch.

3 17

HCI Fresh Deploy - Cannot upgrade 4.4. to 4.5 or add more hosts
by penguin pages 08 Mar '21

08 Mar '21

I reinstalled OS all nodes CentoOS 8 streams. installed cockpit engine and ran through HCI deploy wizard with gluster. this deployed 4.4 version of engine.. then said 4.5 version available. I try to add hosts and get error "Error while executing action: Server thor.penguinpages.local is already part of another cluster." (or other node). I ssh into ovirt engine host and attempt to ssh from that host to three nodes of HCI cluster and it works fine. I added ssh keys for passwordless login but no change I change cluster then datacenter to 4.5. Then attempt to run upgrade but get same error. Seems like chicken / egg issue. I can't get nodes. I can't upgrade 4.4. to 4.5 because I have no nodes to allow maintenance mode. And engine shows "no updates" needed [root@ovirte01 ~]# yum update Last metadata expiration check: 1:39:16 ago on Mon 08 Mar 2021 08:36:40 AM EST. Dependencies resolved. Nothing to do. Complete! Suggestions?

1 2

[security-updates] oVirt node
by Thiago Linhares 08 Mar '21

08 Mar '21

Hello there, I wonder whats the right approach to get security updates for ovirt nodes? (installed using ovirt node iso image) Eg.: 'sudo' package has a know vulnerability until version sudo-1.8.23-9.el7.x86_64. When trying to update this package on a ovirt node, it would not. Checking versionlock, I confirmed it is listed there: # grep sudo /etc/yum/pluginconf.d/versionlock.list 0:libsss_sudo-1.16.4-37.el7_8.3.x86_64 0:sudo-1.8.23-9.el7.x86_64 Regards,

2 1

CVE-2021-3156 && ovirt-node-ng 4.3 && 4.4 (sudo)
by Renaud RAKOTOMALALA 08 Mar '21

08 Mar '21

Hello everyone, I operate several oVirt clusters including pre-productions using ovirt-node-ng images. For our traditional clusters we manage the incident in a unitary way with a dedicated rpm, however for ovirt-node-ng I am not yet up to date with critical package updates process. Do you have any advice or tips? Nice day, Renaud

3 2

Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Marko Vrgotic 08 Mar '21

08 Mar '21

The broker log, these lines are pretty much repeating: MainThread::WARNING::2021-03-03 09:19:12,086::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None' MainThread::INFO::2021-03-03 09:19:12,829::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started MainThread::INFO::2021-03-03 09:19:12,829::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/sub monitors MainThread::INFO::2021-03-03 09:19:12,829::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2021-03-03 09:19:12,832::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain MainThread::INFO::2021-03-03 09:19:12,833::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2021-03-03 09:19:12,834::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mem-free MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor mgmt-bridge MainThread::INFO::2021-03-03 09:19:12,835::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor network MainThread::INFO::2021-03-03 09:19:12,836::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor storage-domain MainThread::INFO::2021-03-03 09:19:12,836::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::WARNING::2021-03-03 09:19:12,836::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be 'None' MainThread::INFO::2021-03-03 09:19:13,574::broker::47::ovirt_hosted_engine_ha.broker.broker.Broker::(run) ovirt-hosted-engine-ha broker 2.3.6 started MainThread::INFO::2021-03-03 09:19:13,575::monitor::40::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Searching for submonitors in /usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/submonitors MainThread::INFO::2021-03-03 09:19:13,575::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load MainThread::INFO::2021-03-03 09:19:13,577::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor cpu-load-no-engine MainThread::INFO::2021-03-03 09:19:13,578::monitor::49::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Loaded submonitor engine-health ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 8 March 2021 at 15:34 To: Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue Hi Didi, Please find the attached logs from Host and Engine. Host ovirt-sj-02 HE Undeploy 2021-03-08 14:15:52 till 2021-03-08 14:18:24 Host ovirt-sj-02 HE Deploy 2021-03-08 14:20:51 till 2021-03-08 14:23:22 I do see errors in the agent and broker and vdsm, but I do not see why it happened. Thank you for helping, let me know if any additional files are needed. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Monday, 8 March 2021 at 09:25 To: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** Hi, On Mon, Mar 8, 2021 at 10:13 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote: > > I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before. > > No errors, found int the deployment, but it seems half done, based on messages I sent in previous email. Please check/share all relevant logs. Thanks. Can be all of /var/log from engine and hosts, and at least: /var/log/ovirt-engine/engine.log /var/log/vdsm/* /var/log/ovirt-hosted-engine-ha/* Best regards, -- Didi

1 0

Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Marko Vrgotic 08 Mar '21

08 Mar '21

Hi Didi, Please find the attached logs from Host and Engine. Host ovirt-sj-02 HE Undeploy 2021-03-08 14:15:52 till 2021-03-08 14:18:24 Host ovirt-sj-02 HE Deploy 2021-03-08 14:20:51 till 2021-03-08 14:23:22 I do see errors in the agent and broker and vdsm, but I do not see why it happened. Thank you for helping, let me know if any additional files are needed. I will send the broker.log separately ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 8 March 2021 at 15:29 To: Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue Hi Didi, Please find the attached logs from Host and Engine. Host ovirt-sj-02 HE Undeploy 2021-03-08 14:15:52 till 2021-03-08 14:18:24 Host ovirt-sj-02 HE Deploy 2021-03-08 14:20:51 till 2021-03-08 14:23:22 I do see errors in the agent and broker and vdsm, but I do not see why it happened. Thank you for helping, let me know if any additional files are needed. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Monday, 8 March 2021 at 09:25 To: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** Hi, On Mon, Mar 8, 2021 at 10:13 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote: > > I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before. > > No errors, found int the deployment, but it seems half done, based on messages I sent in previous email. Please check/share all relevant logs. Thanks. Can be all of /var/log from engine and hosts, and at least: /var/log/ovirt-engine/engine.log /var/log/vdsm/* /var/log/ovirt-hosted-engine-ha/* Best regards, -- Didi

1 0

sanlock error
by kim.kargaard＠noroff.no 08 Mar '21

08 Mar '21

Hi, We are running ovirt 4.3 and whenever we create a new VM, we get the error: HSMGetAllTasksStatusesVDS failed: Error creating a new volume: (u"Volume creation 77be733a-a38a-4125-a3ae-beea960d9e28 failed: (28, 'Sanlock resource write failure', 'No space left on device')",) 3/8/2111:56:57 AM I have checked the engine logs and see the following: 2021-03-08 12:09:18,306+01 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [] BaseAsyncTask::logEndTaskFailure: Task 'd4aa81a1-9adc-45c9-be46-05bc63862f4e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: -- Result: 'cleanSuccess' -- Message: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Volume does not exist: (u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201', -- Exception: 'VDSGenericException: VDSErrorException: Failed to HSMGetAllTasksStatusesVDS, error = Volume does not exist: (u'46bf0806-8d16-40db-92df-8436f43628a4',), code = 201' Not sure how to solve this issue. I have tried restarting the engine, but that did not solve it. Any tips? Thanks.

3 4

Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Yedidyah Bar David 08 Mar '21

08 Mar '21

Hi, On Mon, Mar 8, 2021 at 10:13 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote: > > I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before. > > No errors, found int the deployment, but it seems half done, based on messages I sent in previous email. Please check/share all relevant logs. Thanks. Can be all of /var/log from engine and hosts, and at least: /var/log/ovirt-engine/engine.log /var/log/vdsm/* /var/log/ovirt-hosted-engine-ha/* Best regards, -- Didi

1 0

Re: Upgrade from 4.3.5 to 4.3.10 HE Host issue
by Marko Vrgotic 08 Mar '21

08 Mar '21

I cannot find the reason why the re-Deployment on this Hosts fails, as it was already deployed on it before. No errors, found int the deployment, but it seems half done, based on messages I sent in previous email. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Vrgotic, Marko <M.Vrgotic(a)activevideo.com> Date: Wednesday, 3 March 2021 at 13:12 To: users(a)ovirt.org <users(a)ovirt.org> Subject: Upgrade from 4.3.5 to 4.3.10 HE Host issue Hi oVirt, Yesterday I upgraded one of my oVIrt platform from 4.3.5 to 4.3.10. Platform has 10Hosts with 3Hosts as Hosts for HA. Before upgrade, all three Hosts were acknowledged Hosts for HE (little happy crown). Pre upgrade: Host1 – HE available host Host2 – HE current host Host3 – HE available host Upgraded first HE to 4.3.10, than started with HE Hosts – post upgrade of Host2, the crown disappeared and did not show up. This happens sometimes, and I usually reinstall the host, than execute HE UNDEPLOY and DEPLOY, which fixes the state. However, this time, after such actions, the server is still not an acknowledged HE host – no crown. However, if I try to remove it, oVIrt says that looks like HE is deploy on it and need to be removed, but command “hosted-engine –vm-status” says you must deploy first. I removed the Host, reinstalled the OS on it, deployed it again as new, synced all networks, than deploy once more with HE Deploy, still same issue. Any suggestions? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

4.4.4 Image Copying / Template Create Fails - No Such File
by Alex McWhirter 08 Mar '21

08 Mar '21

I've been wrestling with this all night, digging through various bits of VDSM code trying to figure why and how this is happening. I need to make some templates, but i simply can't. VDSM <host> command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed: ("Command ['/usr/bin/qemu-img', 'convert', '-p', '-t', 'none', '-T', 'none', '-f', 'raw', '-O', 'raw', '/rhev/data-center/mnt/glusterSD/<gluster>:_Temp/45740f16-b3c9-4bb5-ba5f-3e64657fb663/images/6f87b073-c4ec-42f2-87da-d1cb6a08a150/f2dfb779-b49c-4ec9-86cd-741f3fe5b781', '/rhev/data-center/mnt/glusterSD/<gluster>:_Temp/45740f16-b3c9-4bb5-ba5f-3e64657fb663/images/84e56da6-8c26-4518-80a4-20bc395214db/039c5ada-ad6d-45c0-8393-bd4db0bbc366'] failed with rc=1 out=b'' err=bytearray(b'qemu-img: error while writing at byte 738197504: No such file or directory\\n')",) abortedcode=261 3/7/21 5:44:44 AM Following the VDSM logs, i can see the new image gets created, permissions set, etc... but as soon qemu-img starts, it fails like this. I updated all hosts and the engine, rebooted the entire stack, to no avail. So i detached the storage domain, and wiped every host and fresh installed both engine and all nodes, imported the storage domain, and still no dice. Storage domain is gluster volume, single node, created in ovirt. It happens when i make a template, copy an image, or make a new vm from a template. I can still create new vms from blank, and upload images via the web ui. Watching the gluster share, i can see the image being created, but its deleted at some point. I appears to not be being deleted by the template / copying process, as immediately after the above error, i get this one. VDSM command DeleteImageGroupVDS failed: Image does not exist in domain: 'image=4f359545-01a8-439b-832b-18c26194b066, domain=b4507449-ac40-4e35-be66-56441bb696ac' 3/7/21 5:44:44 AM I thought maybe garbage collection, but don't see any indication of that in the logs. Any ideas? I redacted host names from the log output.

3 5

Which Operating System to choose
by sinan＠turka.nl 07 Mar '21

07 Mar '21

I want to set up a small cluster consisting of 2 Hypervisor nodes (self-hosted engine) with shared storage (SAN, ISCSI). According to https://www.ovirt.org/download/ the following Operating Systems are supported with the latest oVirt version (4.4.4): - Red Hat Enterprise Linux 8.3 - CentOS Linux 8.3 - CentOS Stream (Tech Preview) My doubts are: - RHEL 8.3 is free to use up tp 16 hosts, but until when? Red Hat can decide at anytime to change their subscription program. - CentOS Linux is no more, support will stop at the end of 2021. - CentOS Stream is in Tech Preview. I don't like running production environments on "tech preview / beta" versions. I would say CentOS 7, it will receive maintenance updates up to 2024, and switch to RHEL in 2024 if the free-to-use program is still active. Unfortunately, CentOS 7 seems not to be supported. On what Operating System are you running oVirt 4.4.4 and why? Any feedback and advice is welcome. Thanks!

2 1

Cold Move VM Storage failing
by dmmailing010203＠gmail.com 07 Mar '21

07 Mar '21

Morning oVirt. I've rebuilt my oVirt environment and I'm currently on 4.4.5.8-1.el8. I have storage presented via NFS. I'm unable to move / copy VM disks when powered off. However I can Live migrate the disks when powered on. I have three storage domains presented and this is repeated across all three no matter where the storage is located. The disks are thin provisioned. I've tried deactivating and detaching the disk and it's the same NullPointerException error. 2021-03-06 15:13:17,010Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-84) [af63add0-0ef3-4049-8461-b564abedba02] Command 'org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand' failed: null 2021-03-06 15:13:17,010Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.CopyDataCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-84) [af63add0-0ef3-4049-8461-b564abedba02] Exception: java.lang.NullPointerException Engine log with stack trace is at https://textuploader.com/1820f Vdsm log is at https://textuploader.com/1820z I'm just checking I've not missed something obvious before raising a bug if neccessary

2 2

Gluster version upgrades
by Sketch 07 Mar '21

07 Mar '21

Is the gluster version on an oVirt host tied to the oVirt version, or would it be safe to upgrade to newer versions of gluster? I have noticed gluster is often updated to new major versions on oVirt point release upgrades. We have some compute+storage hosts on 4.3.6 which can't be upgraded easily at the moment, but we are having some gluster issues that appear to be due to bugs that I wonder if upgrading might help. Would an in-place upgrade of gluster be a bad idea without also updating oVirt?

2 1