- Users - oVirt List Archives

oVirt Monitoring Alerts via Grafana
by Aviv Litman 14 Sep '21

14 Sep '21

Hi all, Hope you all are doing well. Checkout this new oVirt blog: oVirt Monitoring Alerts via Grafana <https://blogs.ovirt.org/2021/08/ovirt-monitoring-alerts-via-grafana/>. The blog explains how to configure alerts in Grafana for your oVirt environment and provides an example alerts dashboard that you can import, use and edit to your needs. When using alerts significant or critical data changes can be immediately recognized, so don't miss this opportunity to learn how to configure and use this important tool. Feedback, comments and suggestions are more than welcome! -- Aviv Litman BI Associate Software Engineer Red Hat <https://www.redhat.com/> alitman(a)redhat.com <https://www.redhat.com/>

3 3

Firewalld problem
by Jorge Visentini 13 Sep '21

13 Sep '21

Hi all. Anyone with the problem below? *firewalld[29746]: ERROR: COMMAND_FAILED: 'python-nftables' failed: internal:0:0-0: Error: No such file or directory internal:0:0-0: Error: No such file or directory internal:0:0-0: Error: No such file or directory internal:0:0-0: Error: No such file or directory internal:0:0-0: Error: No such file or directory internal:0:0-0: Error: No such file or directory internal:0:0-0: Error: No such file or directory internal:0:0-0: Error: No such file or directory* I'm using ovirt 4.4.8.3. The python-nftables package doesn't exist on the repository, just the python3-nftables package... Thanks!! -- Att, Jorge Visentini +55 55 98432-9868

1 0

about the cloud-init options on the vm portal
by Tommy Sway 13 Sep '21

13 Sep '21

It seems that the options for cloud-init in the VM portal interface are only host name and SSH key, other options that can be seen in the administrator interface are not visible. Is there a solution to this problem? Thanks!

1 0

Cannot Start VM After Pausing due to Storage I/O Error
by bob.franzke＠mdaemon.com 13 Sep '21

13 Sep '21

Noticed I had a VM that was 'paused' due to a 'Storage I/O error. I inherited this system from another admin and have no idea where to start figuring this out. We have a 4-node Ovirt cluster with a 5 Manager node. The VM in question is running on a host vm-host-colo-4. Best I can tell the VMs run on a gluster replicated volume replicated between all 4 nodes, with node 1 running as an arbiter node for the gluster volume. Other VMs are running on this host 4 so not sure what the issue is with this one VM. When I look at the status of the gluster volume for this host, I see the self-heal info for the bricks is listed as 'N/A' for this host. All the other hosts in the cluster list this info as 'OK'. When I cd into the gluster directory on host 4, I don't see the same things as I do on the other hosts. I am not sure this is an issue but its just different. When running various gluster commands gluster seems to respond. See below: [root@vm-host-colo-4 gluster]# gluster volume info all Volume Name: gl-colo-1 Type: Replicate Volume ID: 2c545e19-9468-487e-9e9b-cd3202fc24c4 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: 10.20.101.181:/gluster/gl-colo-1/brick1 Brick2: 10.20.101.183:/gluster/gl-colo-1/brick1 Brick3: 10.20.101.185:/gluster/gl-colo-1/brick1 (arbiter) Options Reconfigured: network.ping-timeout: 30 cluster.granular-entry-heal: enable performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 cluster.choose-local: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off transport.address-family: inet nfs.disable: on performance.client-io-threads: off Volume Name: gl-vm-host-4 Type: Distribute Volume ID: a2ba6b29-2366-4a7e-bda8-2e0574cf4afa Status: Started Snapshot Count: 0 Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.20.101.187:/gluster/gl-vm-host-colo-4 Options Reconfigured: network.ping-timeout: 30 cluster.granular-entry-heal: enable network.remote-dio: off performance.strict-o-direct: on storage.owner-gid: 36 storage.owner-uid: 36 auth.allow: * user.cifs: disable transport.address-family: inet nfs.disable: on [root@vm-host-colo-4 gluster]# [root@vm-host-colo-4 gluster]# gluster-eventsapi status Webhooks: http://mydesktop.altn.int:80/ovirt-engine/services/glusterevents +-------------------------+-------------+-----------------------+ | NODE | NODE STATUS | GLUSTEREVENTSD STATUS | +-------------------------+-------------+-----------------------+ | vm-host-colo-1.altn.int | UP | OK | | vm-idev-colo-1.altn.int | UP | OK | | vm-host-colo-2.altn.int | UP | OK | | localhost | UP | OK | +-------------------------+-------------+-----------------------+ [root@vm-host-colo-4 gluster]# gluster volume status gl-vm-host-4 Status of volume: gl-vm-host-4 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.20.101.187:/gluster/gl-vm-host-col o-4 49152 0 Y 33221 Task Status of Volume gl-vm-host-4 ------------------------------------------------------------------------------ There are no active volume tasks I also get a timeout error when doing a 'gluster volume status' on this node. So while some aspects seem fine with the gluster volume, some don't. Should I restart the glusterd daemon or will that mess things up? I am not sure if this is due to something wrong with the gluster volume or with the vm-host's ability to access the data for the VM disk, meaning a true I/O problem. There are two VMs in this state both running on this host and I am not sure how to proceed to get them running again. Should I force this VM to be on a different host by editing the VM or should I try and make it work on the host its on. As mentioned, many other VMs are running on this host so not sure why these two have an issue. Up front apologies here. I am a network engineer and not a VM/Ovirt expert. This was dropped in my lap due to a layoff and could use some help on where to go from here. Thanks in advance for any help.

5 7

Create template from snapshot of vm using MBS disk
by ssarang520＠gmail.com 13 Sep '21

13 Sep '21

Hi, If I create a template with a snapshot of the vm that uses the mbs disk, the template cannot be used to create a new vm. Is this normal or a bug?

2 6

problems testing 4.3.10 to 4.4.8 upgrade SHE
by Gianluca Cecchi 13 Sep '21

13 Sep '21

Hello, I'm testing what in object in a test env with novirt1 and novirt2 as hosts. First reinstalled host is novirt2 For this I downloaded the 4.4.8 iso of the node: https://resources.ovirt.org/pub/ovirt-4.4/iso/ovirt-node-ng-installer/4.4.8… before running the restore command for the first scratched node I pre-installed the appliance rpm on it and I got: ovirt-engine-appliance-4.4-20210818155544.1.el8.x86_64 I selected to pause an d I arrived here with local vm engine completing its setup: INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Add host] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Include after_add_host tasks files] [ INFO ] You can now connect to https://novirt2.localdomain.local:6900/ovirt-engine/ and check the status of this host and eventually remediate it, please continue only when the host is listed as 'up' [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Create temporary lock file] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Pause execution until /tmp/ansible.4_o6a2wo_he_setup_lock is removed, delete it once ready to proceed] But connecting t the provided https://novirt2.localdomain.local:6900/ovirt-engine/ url I see that only the still 4.3.10 host results up while novirt2 is not resp[onsive vm situation: https://drive.google.com/file/d/1OwHHzK0owU2HWZqvHFaLLbHVvjnBhRRX/view?usp=… storage situation: https://drive.google.com/file/d/1D-rmlpGsKfRRmYx2avBk_EYCG7XWMXNq/view?usp=… hosts situation: https://drive.google.com/file/d/1yrmfYF6hJFzKaG54Xk0Rhe2kY-TIcUvA/view?usp=… In engine.log I see 2021-08-25 09:14:38,548+02 ERROR [org.ovirt.engine.core.vdsbroker.HostDevListByCapsVDSCommand] (EE-ManagedThreadFactory-engine-Thread-4) [5f4541ee] Command 'HostDevListByCapsVDSCommand(HostName = novirt2.localdomain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='ca9ff6f7-5a7c-4168-9632-998c52f76cfa', vds='Host[novirt2.localdomain.local,ca9ff6f7-5a7c-4168-9632-998c52f76cfa]'})' execution failed: java.net.ConnectException: Connection refused and continuouslly this message... I also tried to restart vdsmd on novit2 but nothing changed. Do I have to restart the HA daemons on novirt2? Any insight? Thanks Gianluca

3 15

Error when trying to change master storage domain
by Matthew Benstead 11 Sep '21

11 Sep '21

Hello, I'm trying to decommission the old master storage domain in ovirt, and replace it with a new one. All of the VMs have been migrated off of the old master, and everything has been running on the new storage domain for a couple months. But when I try to put the old domain into maintenance mode I get an error. Old Master: vm-storage-ssd New Domain: vm-storage-ssd2 The error is: Failed to Reconstruct Master Domain for Data Center EDC2 As well as: Sync Error on Master Domain between Host daccs01 and oVirt Engine. Domain: vm-storage-ssd is marked as Master in oVirt Engine database but not on the Storage side. Please consult with Support on how to fix this issue. 2021-07-28 11:41:34,870-07 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engine-Thread-23) [] Master domain version is not in sync between DB and VDSM. Domain vm-storage-ssd marked as master, but the version in DB: 283 and in VDSM: 280 And: Not stopping SPM on vds daccs01, pool id f72ec125-69a1-4c1b-a5e1-313fcb70b6ff as there are uncleared tasks Task '5fa9edf0-56c3-40e4-9327-47bf7764d28d', status 'finished' After a couple minutes all the domains are marked as active again and things continue, but vm-storage-ssd is still listed as the master domain. Any thoughts? This is on 4.3.10.4-1.el7 on CentOS 7. engine=# SELECT storage_name, storage_pool_id, storage, status FROM storage_pool_with_storage_domain ORDER BY storage_name; storage_name | storage_pool_id | storage | status -----------------------+--------------------------------------+----------------------------------------+-------- compute1-iscsi-ssd | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | yvUESE-yWUv-VIWL-qX90-aAq7-gK0I-EqppRL | 1 compute7-iscsi-ssd | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | 8ekHdv-u0RJ-B0FO-LUUK-wDWs-iaxb-sh3W3J | 1 export-domain-storage | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | d3932528-6844-481a-bfed-542872ace9e5 | 1 iso-storage | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | f800b7a6-6a0c-4560-8476-2f294412d87d | 1 vm-storage-7200rpm | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | a0bff472-1348-4302-a5c7-f1177efa45a9 | 1 vm-storage-ssd | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | 95acd9a4-a6fb-4208-80dd-1c53d6aacad0 | 1 vm-storage-ssd2 | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff | 829d0600-c3f7-4dae-a749-d7f05c6a6ca4 | 1 (7 rows) Thanks, -Matthew --

2 12

Poor gluster performances over 10Gbps network
by Mathieu Valois 10 Sep '21

10 Sep '21

Sorry for double post but I don't know if this mail has been received. Hello everyone, I know this issue was already treated on this mailing list. However none of the proposed solutions is satisfying me. Here is my situation : I've got 3 hyperconverged gluster ovirt nodes, with 6 network interfaces, bounded in bunches of 2 (management, VMs and gluster). The gluster network is on a dedicated bound where the 2 interfaces are directly connected to the 2 other ovirt nodes. Gluster is apparently using it : # gluster volume status vmstore Status of volume: vmstore Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick gluster-ov1:/gluster_bricks /vmstore/vmstore 49152 0 Y 3019 Brick gluster-ov2:/gluster_bricks /vmstore/vmstore 49152 0 Y 3009 Brick gluster-ov3:/gluster_bricks /vmstore/vmstore where 'gluster-ov{1,2,3}' are domain names referencing nodes in the gluster network. This networks has 10Gbps capabilities : # iperf3 -c gluster-ov3 Connecting to host gluster-ov3, port 5201 [ 5] local 10.20.0.50 port 46220 connected to 10.20.0.51 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.16 GBytes 9.92 Gbits/sec 17 900 KBytes [ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 900 KBytes [ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec 4 996 KBytes [ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec 1 996 KBytes [ 5] 4.00-5.00 sec 1.15 GBytes 9.89 Gbits/sec 0 996 KBytes [ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 7.00-8.00 sec 1.15 GBytes 9.91 Gbits/sec 0 996 KBytes [ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes [ 5] 9.00-10.00 sec 1.15 GBytes 9.90 Gbits/sec 0 996 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 GBytes 9.90 Gbits/sec 22 sender [ 5] 0.00-10.04 sec 11.5 GBytes 9.86 Gbits/sec receiver iperf Done. However, VMs stored on the vmstore gluster volume has poor write performances, oscillating between 100KBps and 30MBps. I almost always observe a write spike (180Mbps) at the beginning until around 500MB written, then it drastically falls at 10MBps, sometimes even less (100KBps). Hypervisors have 32 threads (2 sockets, 8 cores per socket, 2 threads per core). Here is the volume settings : Volume Name: vmstore Type: Replicate Volume ID: XXX Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gluster-ov1:/gluster_bricks/vmstore/vmstore Brick2: gluster-ov2:/gluster_bricks/vmstore/vmstore Brick3: gluster-ov3:/gluster_bricks/vmstore/vmstore Options Reconfigured: performance.io-thread-count: 32 # was 16 by default. cluster.granular-entry-heal: enable storage.owner-gid: 36 storage.owner-uid: 36 cluster.lookup-optimize: off server.keepalive-count: 5 server.keepalive-interval: 2 server.keepalive-time: 10 server.tcp-user-timeout: 20 network.ping-timeout: 30 server.event-threads: 4 client.event-threads: 8 # was 4 by default cluster.choose-local: off features.shard: on cluster.shd-wait-qlength: 10000 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable performance.strict-o-direct: on network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off auth.allow: * user.cifs: off storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on performance.client-io-threads: on When I naively write directly on the logical volume, which is mounted on a material RAID5 3-disks array, I have interesting performances: # dd if=/dev/zero of=a bs=4M count=2048 2048+0 records in 2048+0 records out 8589934592 bytes (8.6 GB, 8.0 GiB) copied, 17.2485 s, 498 MB/s #urandom gives around 200MBps Moreover, hypervisors have SSD which have been configured as lvcache, but I'm unsure how to test it efficiently. I can't find where is the problem, as every piece of the chain is apparently doing well ... Thanks anyone for helping me :) -- téïcée <https://www.teicee.com/?pk_campaign=Email> *Mathieu Valois* Bureau Caen: Quartier Kœnig - 153, rue Géraldine MOCK - 14760 Bretteville-sur-Odon Bureau Vitré: Zone de la baratière - 12, route de Domalain - 35500 Vitré 02 72 34 13 20 | www.teicee.com <https://www.teicee.com/?pk_campaign=Email> téïcée sur facebook <https://www.facebook.com/teicee> téïcée sur twitter <https://twitter.com/Teicee_fr> téïcée sur linkedin <https://www.linkedin.com/company/t-c-e> téïcée sur viadeo <https://fr.viadeo.com/fr/company/teicee> Datadocké

5 8

about the certification
by CuiTao 09 Sep '21

09 Sep '21

I wanted to understand how certificates work in OLVM, including the following: What certificates exist in the OLVM system? Second, what are the expiration dates of these certificates? Third, how to view the remaining time of the certificate and regenerate the certificate before expiration? Thanks!

1 0

Redeploying hosted engine from backup
by Artem Tambovskiy 09 Sep '21

09 Sep '21

Hello, Just had an issue with my cluster with a self-hosted engine (hosted-engine is not coming up) and decided to redeploy it as I have a backup. Just tried a hosted-engine --deploy --restore-from-file=engine.backup But the script is asking questions like DC name, Cluster name which I can't recall correctly. What will be the consequence of the wrong answer? Is there any chance to get this info from the hosts or backup file? -- Regards, Artem

2 1