- Users - oVirt List Archives

Lost space in /var/log
by Andrei Verovski 23 Jun '22

23 Jun '22

Hi ! I have strange situation with low disk space on /var/log/, yet I can't figure out what have consumed so much space. Look at the du output. Thanks in advance for any suggestion(s). # df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-var 15G 1.3G 13G 9% /var /dev/mapper/centos-var_log 7.8G 7.1G 293M 97% /var/log /dev/mapper/centos-var_log_audit 2.0G 41M 1.8G 3% /var/log/audit # # du -ah /var/log | sort -n -r | head -n 20 664K /var/log/vdsm 660K /var/log/vdsm/vdsm.log 624K /var/log/gdm 580K /var/log/anaconda/storage.log 480K /var/log/anaconda/packaging.log 380K /var/log/gdm/:0.log.4 316K /var/log/anaconda/syslog 276K /var/log/tuned 252K /var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806 220K /var/log/Xorg.0.log 168K /var/log/gdm/:0.log 156K /var/log/yum.log-20191117 156K /var/log/secure-20180726 132K /var/log/libvirt/qemu/NextCloud-Collabora-Active.log 128K /var/log/anaconda/anaconda.log 120K /var/log/hp-snmp-agents 116K /var/log/hp-snmp-agents/cma.log 112K /var/log/vinchin 104K /var/log/vinchin/kvm_backup_service 104K /var/log/tuned/tuned.log.2 # # find . -printf '%s %p\n'| sort -nr | head -10 16311686 ./rhsm/rhsm.log 4070423 ./cron-20180726 3667146 ./anaconda/journal.log 3409071 ./secure 3066660 ./rhsm/rhsm.log-20180726 2912670 ./audit/audit.log 1418007 ./sanlock.log 1189580 ./vdsm/vdsm.log 592718 ./anaconda/storage.log 487567 ./anaconda/packaging.log

3 4

Re: Lost space in /var/log
by Michael Thomas 22 Jun '22

22 Jun '22

This can also happen with a misconfigured logrotate config. If a process is writing to a large log file, and logrotate comes along and removes it, then the process still has an open filehandle to the large file even though you can't see it. The space won't get removed until the process closes the filehandle (eg rebooting). The following command should give you a list of these ghost files that are still open but have been removed from the directory tree: lsof | grep '(deleted)' Stackexchange has some additional useful tips: https://unix.stackexchange.com/questions/68523/find-and-remove-large-files-… --Mike On 6/22/22 15:49, Matthew.Stier(a)fujitsu.com wrote: > Deleted some files to "clean up" /var/log, but the space was not recovered? > > Space for deleted files are only recovered when all references to that space are removed. This includes the directory reference, and any open file-handles to the file. > > The is a popular trick for a self-deleting scratch file. Create a file, open file-handle to it, and then remove the file. The open file-handle can then be written to, and read from, as long as you like, but when the file-handle is closed, explicitly or implicitly, the space is automatically recovered. > > > -----Original Message----- > From: Andrei Verovski <andreil1(a)starlett.lv> > Sent: Wednesday, June 22, 2022 3:27 PM > To: users(a)ovirt.org > Subject: [ovirt-users] Lost space in /var/log > > Hi ! > > I have strange situation with low disk space on /var/log/, yet I can't figure out what have consumed so much space. > Look at the du output. > > Thanks in advance for any suggestion(s). > > > # df -h > > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/centos-var 15G 1.3G 13G 9% /var > /dev/mapper/centos-var_log 7.8G 7.1G 293M 97% /var/log > /dev/mapper/centos-var_log_audit 2.0G 41M 1.8G 3% /var/log/audit > > # > # du -ah /var/log | sort -n -r | head -n 20 > > 664K /var/log/vdsm > 660K /var/log/vdsm/vdsm.log > 624K /var/log/gdm > 580K /var/log/anaconda/storage.log > 480K /var/log/anaconda/packaging.log > 380K /var/log/gdm/:0.log.4 > 316K /var/log/anaconda/syslog > 276K /var/log/tuned > 252K /var/log/libvirt/qemu/NextCloud-Collabora-LVM.log-20210806 > 220K /var/log/Xorg.0.log > 168K /var/log/gdm/:0.log > 156K /var/log/yum.log-20191117 > 156K /var/log/secure-20180726 > 132K /var/log/libvirt/qemu/NextCloud-Collabora-Active.log > 128K /var/log/anaconda/anaconda.log > 120K /var/log/hp-snmp-agents > 116K /var/log/hp-snmp-agents/cma.log > 112K /var/log/vinchin > 104K /var/log/vinchin/kvm_backup_service > 104K /var/log/tuned/tuned.log.2 > > # > # find . -printf '%s %p\n'| sort -nr | head -10 > > 16311686 ./rhsm/rhsm.log > 4070423 ./cron-20180726 > 3667146 ./anaconda/journal.log > 3409071 ./secure > 3066660 ./rhsm/rhsm.log-20180726 > 2912670 ./audit/audit.log > 1418007 ./sanlock.log > 1189580 ./vdsm/vdsm.log > 592718 ./anaconda/storage.log > 487567 ./anaconda/packaging.log > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/N566CST3DAXIL… > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JCAQU32UZSEWK…

1 0

Q: Error ID 119 - ioctl(KVM_CREATE_VM) failed: Cannot allocate memory
by Andrei Verovski 22 Jun '22

22 Jun '22

Hi ! Does this error message means node runs out of RAM? Quite strange since there are about 20 GB of free RAN, and VM requires only 8. I noticed this happen till Vinchin was in a process of removing snapshot from another VM. Thanks. VM Jumis_VM-Active is down with error. Exit message: internal error: process exited while connecting to monitor: ioctl(KVM_CREATE_VM) failed: 12 Cannot allocate memory 2022-06-22T19:50:46.501704Z qemu-kvm: failed to initialize KVM: Cannot allocate memory.

1 0

Move Master Storage Domain
by john＠penrod.net 22 Jun '22

22 Jun '22

Running oVirt 4.3.10. Three storage domains. #1 is the current master. No disks/data remain on it. All virtual machines have been moved. I plan to put it in maintenance mode and let it force an election. I don't care which of the remaining two become master. Will this impact virtual machines that are currently running? If possible, I need to keep them running. Thoughts?

1 0

Cannot export VM to another data domain. "o.original_template is undefined"
by Gilboa Davara 22 Jun '22

22 Jun '22

Hello, 4.4.10 Gluster based cluster w/ 3 nodes. I'm backing up all VMs before upgrading the setup to 4.5. (Hopefully done right before 4.5.1 w/ Gluster fix is released). When trying to export a VM to another data domain, several VMs show the following error: "Export VM Failed. o.original_template is undefined". Engine log looks clean (I see nothing about the failed upgrade). Any idea what's broken? - Gilboa

2 11

datacenter, node becam non responsive after power cut
by moulayhakim3＠yahoo.com 22 Jun '22

22 Jun '22

hello team, i have self hosted engine, after power cut , the datacenter and node becam non responsive, can you help me please

1 0

Gluster Volume cannot be activated Ovirt 4.5 Centos 8 Stream
by m.rohweder＠itm-h.de 21 Jun '22

21 Jun '22

Hi, i convertet my ovirt 4.5 tu ovirt 4.5 hyperconverged. (activate on cluster the gluster service and reinstalled all hosts) I can create bricks on hosts and i'm able to create a Volume. All with the Ovirt GUI. But if i want to activate the Volume, i get the error mesage that no host with running gluster is found in my cluster. All hosts show message that ovirt think glusterd ist not running, but its running on all hosts. What can i du to use gluster with the lokal storage on each host? Greatings Michael

2 2

Ovirt 4.4 - starting just-installed host from ovirt console fails
by David Johnson 21 Jun '22

21 Jun '22

Good afternoon all, I recently had to rebuild my cluster due to a self inflicted error. I have finally managed to get the ovirt host software installed and communicating on all hosts. The first host installed and started cleanly. However, after installation the second host is failing to start. Prior to my cluster crash, this host was running well in the cluster. During the downtime, we applied microcode and BIOS updates as part of the recovery process. I have reviewed this chain: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/N3PPT34GBRLPLT… and reached a dead end. Based on what I see (in the long stream of logs and info following), it looks like I should change the cluster CPU type from Cascadelake to Haswell to restore normal operation. The long involved stuff: The Engine reports: [image: image.png] Host CPU type is not compatible with Cluster Properties. [image: image.png] The host CPU does not match the Cluster CPU Type and is running in a degraded mode. It is missing the following CPU flags: model_Cascadelake-Server-noTSX. Please update the host CPU microcode or change the Cluster CPU Type. The Cluster definition is: [image: image.png] *lscpu returns:* Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz BIOS Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz Stepping: 2 CPU MHz: 3300.000 CPU max MHz: 3300.0000 CPU min MHz: 1200.0000 BogoMIPS: 4988.45 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 30720K NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d *cpuid returns:* CPU 0: vendor_id = "GenuineIntel" version information (1/eax): processor type = primary processor (0) family = 0x6 (6) model = 0xf (15) stepping id = 0x2 (2) extended family = 0x0 (0) extended model = 0x3 (3) (family synth) = 0x6 (6) (model synth) = 0x3f (63) (simple synth) = Intel (unknown type) (Haswell C1/M1/R2) {Haswell}, 22nm *virsh domcapabilities returns:* <domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>kvm</domain> <machine>pc-i440fx-rhel7.6.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <iothreads supported='yes'/> <os supported='yes'> <enum name='firmware'/> <loader supported='yes'> <value>/usr/share/OVMF/OVMF_CODE.secboot.fd</value> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> <enum name='secure'> <value>no</value> </enum> </loader> </os> <cpu> <mode name='host-passthrough' supported='yes'> <enum name='hostPassthroughMigratable'> <value>on</value> <value>off</value> </enum> </mode> <mode name='maximum' supported='yes'> <enum name='maximumMigratable'> <value>on</value> <value>off</value> </enum> </mode> <mode name='host-model' supported='yes'> <model fallback='forbid'>Haswell-noTSX-IBRS</model> <vendor>Intel</vendor> <feature policy='require' name='vme'/> <feature policy='require' name='ss'/> <feature policy='require' name='vmx'/> <feature policy='require' name='pdcm'/> <feature policy='require' name='f16c'/> <feature policy='require' name='rdrand'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='arat'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='umip'/> <feature policy='require' name='md-clear'/> <feature policy='require' name='stibp'/> <feature policy='require' name='arch-capabilities'/> <feature policy='require' name='ssbd'/> <feature policy='require' name='xsaveopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='abm'/> <feature policy='require' name='invtsc'/> <feature policy='require' name='ibpb'/> <feature policy='require' name='ibrs'/> <feature policy='require' name='amd-stibp'/> <feature policy='require' name='amd-ssbd'/> <feature policy='require' name='skip-l1dfl-vmentry'/> <feature policy='require' name='pschange-mc-no'/> </mode> <mode name='custom' supported='yes'> <model usable='yes'>qemu64</model> <model usable='yes'>qemu32</model> <model usable='no'>phenom</model> <model usable='yes'>pentium3</model> <model usable='yes'>pentium2</model> <model usable='yes'>pentium</model> <model usable='yes'>n270</model> <model usable='yes'>kvm64</model> <model usable='yes'>kvm32</model> <model usable='yes'>coreduo</model> <model usable='yes'>core2duo</model> <model usable='no'>athlon</model> <model usable='yes'>Westmere-IBRS</model> <model usable='yes'>Westmere</model> <model usable='no'>Snowridge</model> <model usable='no'>Skylake-Server-noTSX-IBRS</model> <model usable='no'>Skylake-Server-IBRS</model> <model usable='no'>Skylake-Server</model> <model usable='no'>Skylake-Client-noTSX-IBRS</model> <model usable='no'>Skylake-Client-IBRS</model> <model usable='no'>Skylake-Client</model> <model usable='yes'>SandyBridge-IBRS</model> <model usable='yes'>SandyBridge</model> <model usable='yes'>Penryn</model> <model usable='no'>Opteron_G5</model> <model usable='no'>Opteron_G4</model> <model usable='no'>Opteron_G3</model> <model usable='yes'>Opteron_G2</model> <model usable='yes'>Opteron_G1</model> <model usable='yes'>Nehalem-IBRS</model> <model usable='yes'>Nehalem</model> <model usable='yes'>IvyBridge-IBRS</model> <model usable='yes'>IvyBridge</model> <model usable='no'>Icelake-Server-noTSX</model> <model usable='no'>Icelake-Server</model> <model usable='no' deprecated='yes'>Icelake-Client-noTSX</model> <model usable='no' deprecated='yes'>Icelake-Client</model> <model usable='yes'>Haswell-noTSX-IBRS</model> <model usable='yes'>Haswell-noTSX</model> <model usable='no'>Haswell-IBRS</model> <model usable='no'>Haswell</model> <model usable='no'>EPYC-Rome</model> <model usable='no'>EPYC-Milan</model> <model usable='no'>EPYC-IBPB</model> <model usable='no'>EPYC</model> <model usable='no'>Dhyana</model> <model usable='no'>Cooperlake</model> <model usable='yes'>Conroe</model> <model usable='no'>Cascadelake-Server-noTSX</model> <model usable='no'>Cascadelake-Server</model> <model usable='no'>Broadwell-noTSX-IBRS</model> <model usable='no'>Broadwell-noTSX</model> <model usable='no'>Broadwell-IBRS</model> <model usable='no'>Broadwell</model> <model usable='yes'>486</model> </mode> </cpu> <memoryBacking supported='yes'> <enum name='sourceType'> <value>file</value> <value>anonymous</value> <value>memfd</value> </enum> </memoryBacking> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> <value>sata</value> </enum> <enum name='model'> <value>virtio</value> <value>virtio-transitional</value> <value>virtio-non-transitional</value> </enum> </disk> <graphics supported='yes'> <enum name='type'> <value>vnc</value> <value>spice</value> <value>egl-headless</value> </enum> </graphics> <video supported='yes'> <enum name='modelType'> <value>vga</value> <value>cirrus</value> <value>qxl</value> <value>virtio</value> <value>none</value> <value>bochs</value> <value>ramfb</value> </enum> </video> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'/> </hostdev> <rng supported='yes'> <enum name='model'> <value>virtio</value> <value>virtio-transitional</value> <value>virtio-non-transitional</value> </enum> <enum name='backendModel'> <value>random</value> <value>egd</value> <value>builtin</value> </enum> </rng> <filesystem supported='yes'> <enum name='driverType'> <value>path</value> <value>handle</value> <value>virtiofs</value> </enum> </filesystem> <tpm supported='yes'> <enum name='model'> <value>tpm-tis</value> <value>tpm-crb</value> </enum> <enum name='backendModel'> <value>passthrough</value> <value>emulator</value> </enum> </tpm> </devices> <features> <gic supported='no'/> <vmcoreinfo supported='yes'/> <genid supported='yes'/> <backingStoreInput supported='yes'/> <backup supported='yes'/> <sev supported='no'/> </features> </domainCapabilities> Please advise.

1 1

Preferred RHEL Based Distro For oVirt
by Clint Boggio 21 Jun '22

21 Jun '22

Good Day All; i am inquiring about which RHEL based distros are currently preferred and which ones are currently supported. I know the oVirt project is a RH entity and so RHEL and CentOS-Stream are the base offering. Would it, or is it, feasible for Rocky 8.X, or Alma 8.X to be the base OS for an oVirt deployment seeing as though they both RHEL clones ? How confident is the user community in the stability of CentOS-Stream in terms of production use as compared to Alma or Rocky ?

3 4

failed to mount hosted engine gluster storage - how to debug?
by diego.ercolani＠ssis.sm 21 Jun '22

21 Jun '22

Hello, I have an issue probably related to my particular implementation but I think some controls are missing Here is the story. I have a cluster of two nodes 4.4.10.3 with an upgraded kernel as the cpu (Ryzen 5) suffer from an incompatibility issue with the kernel provided by 4.4.10.x series. On each node there are three glusterfs "partitions" in replica mode, one for the hosted_engine, the other two are for user usage. The third node was an old i3 workstation only used to provide the arbiter partition to the glusterfs cluster. I installed a new server (ryzen processor) with 4.5.0 and successfully installed glusterfs 10.1 and inserted the arbiter bricks implemented on glusterfs 10.1 while the replica bricks are 8.6 after removing the old i3 provided bricks. I successfully imported the new node in the ovirt engine (after updating the engine to 4.5) The problem is that the ovirt-ha-broker doesn't start complaining that is not possible to connect the storage. (I suppose the hosted_engine storage) so I did some digs that I'm going to show here: #### 1. The node seem to be correctly configured: [root@ovirt-node3 devices]# vdsm-tool validate-config SUCCESS: ssl configured to true. No conflicts [root@ovirt-node3 devices]# vdsm-tool configure Checking configuration status... libvirt is already configured for vdsm SUCCESS: ssl configured to true. No conflicts sanlock is configured for vdsm Managed volume database is already configured lvm is configured for vdsm Current revision of multipath.conf detected, preserving Running configure... Done configuring modules to VDSM. [root@ovirt-node3 devices]# vdsm-tool validate-config SUCCESS: ssl configured to true. No conflicts #### 2. the node refuses to mount via hosted-engine (same error in broker.log) [root@ovirt-node3 devices]# hosted-engine --connect-storage Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/connect_storage_server.py", line 30, in <module> timeout=ohostedcons.Const.STORAGE_SERVER_TIMEOUT, File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 312, in connect_storage_server sserver.connect_storage_server(timeout=timeout) File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_server.py", line 451, in connect_storage_server 'Connection to storage server failed' RuntimeError: Connection to storage server failed ##### 3. manually mount of glusterfs work correctly [root@ovirt-node3 devices]# grep storage /etc/ovirt-hosted-engine/hosted-engine.conf storage=ovirt-node2.ovirt:/gveng # The following are used only for iSCSI storage [root@ovirt-node3 devices]# [root@ovirt-node3 devices]# mount -t glusterfs ovirt-node2.ovirt:/gveng /mnt/tmp/ [root@ovirt-node3 devices]# ls -l /mnt/tmp total 0 drwxr-xr-x. 6 vdsm kvm 64 Dec 15 19:04 7b8f1cc9-e3de-401f-b97f-8c281ca30482 What else should I control? Thank you and sorry for the long message Diego

5 9

Virtual disk shareable checkbox is greyed out when using GlusterFS storage domains
by Ken Holmes 21 Jun '22

21 Jun '22

Is there a work around that will allow multiple VM's to share a disk with out using the network? I read Dave White's Thread but saw no resolution except maybe using NetApp. Thanks

1 0

moVirt delisted from Google Play
by Filip Krepinsky 21 Jun '22

21 Jun '22

Hi all, Unfortunately, we decided to delist moVirt from Google Play. As you might have noticed, the app has not been maintained for some time and also the main repository https://github.com/oVirt/moVirt has been archived. It is not possible for us to maintain store presence due to new requirements for Google APIs (and thus our libraries that are obsolete at this point), app interoperability and in general to fulfill expectations of our users. For users who still wish to use moVirt, you can keep your current application installed or download an APK ( https://github.com/oVirt/moVirt/releases/tag/v2.1) and install moVirt manually. Just keep in mind that the app might not behave properly (mostly on newer versions of Android). I hope moVirt has been helpful with managing your envs :) Best regards, Filip

1 0

Dedicated Migration Network
by Clint Boggio 21 Jun '22

21 Jun '22

Good Day All; I am in the process of assembling a new oVirt cluster and I am wondering if there is any benefit to having a dedicated set of nics solely dedicated to VM migration. If so, would those nics (bonded) be on their own vlan and subnet or would they share a subnet with gluster storage or the management network ? Right now i'm planning on the following architecture. 1. Management = 10G X2 Bond per hypervisor host (10.66.0.0/24) 2. Gluster/iSCSI Storage = 10G X2 Bond per hypervisor host (10.244.0.0/24) 3. VM Production Network(s) = 10G X2 Bond per hypervisor host (No IP Range) 4. ?? Possible Dedicated VM Migration Thank you very much as any input would be greatly appreciated

4 6

"Retrieval of iSCSI targets failed" during hosted engine deployment on oVirt node 4.5
by pat＠patfruth.com 21 Jun '22

21 Jun '22

I have freshly installed ovirt node 4.5 from the iso download here; https://resources.ovirt.org/pub/ovirt-4.5/iso/ovirt-node-ng-installer/4.5.0… output from 'rpm -qa | grep ovirt' shows ovirt-hosted-engine-setup-2.6.3-1.el8.noarch ovirt-imageio-daemon-2.4.3-1.el8.x86_64 python38-ovirt-engine-sdk4-4.5.1-1.el8.x86_64 ovirt-imageio-common-2.4.3-1.el8.x86_64 ovirt-node-ng-image-update-placeholder-4.5.0.3-1.el8.noarch ovirt-openvswitch-2.15-3.el8.noarch python38-ovirt-imageio-client-2.4.3-1.el8.x86_64 ovirt-openvswitch-ipsec-2.15-3.el8.noarch ovirt-openvswitch-ovn-common-2.15-3.el8.noarch ovirt-openvswitch-ovn-host-2.15-3.el8.noarch centos-release-ovirt45-8.7-1.el8s.noarch ovirt-provider-ovn-driver-1.2.36-1.el8.noarch ovirt-host-dependencies-4.5.0-3.el8.x86_64 ovirt-release-host-node-4.5.0.3-1.el8.x86_64 ovirt-ansible-collection-2.0.3-1.el8.noarch ovirt-vmconsole-1.0.9-1.el8.noarch python3-ovirt-engine-sdk4-4.5.1-1.el8.x86_64 ovirt-node-ng-nodectl-4.4.2-1.el8.noarch ovirt-openvswitch-ovn-2.15-3.el8.noarch ovirt-hosted-engine-ha-2.5.0-1.el8.noarch ovirt-host-4.5.0-3.el8.x86_64 ovirt-engine-appliance-4.5-20220511122240.1.el8.x86_64 ovirt-vmconsole-host-1.0.9-1.el8.noarch ovirt-python-openvswitch-2.15-3.el8.noarch python38-ovirt-imageio-common-2.4.3-1.el8.x86_64 python3-ovirt-node-ng-nodectl-4.4.2-1.el8.noarch cockpit-ovirt-dashboard-0.16.0-1.el8.noarch python3-ovirt-setup-lib-1.3.3-1.el8.noarch ovirt-imageio-client-2.4.3-1.el8.x86_64 During the hosted engine deployment process, I get thru Step 3 ( the "Prepare VM" step) successfully. On Step 4 (Storage Settings), I set; - Storage type = iSCSI - Portal IP address = my ISCSI target's ip address - Accept the default Portal port number, which is already set to 3260 - Leave username & password blank (as I have no CHAP configured on the ISCSI target system) When I click the "Retrieve Target List" button there is a brief pause, followed by red error message which says "Retrieval of iSCSI targets failed" Upon reviewing the files in /var/log/ovirt-hosted-engine-setup on the ovirt node, I find a new log file named ovirt-hosted-engine-setup-ansible-iscsi_discover-20220614084053-in517x.log The message near the end of the log file are as follows; ------ snip ------ ..... 2022-06-14 08:41:03,510-0600 INFO ansible task start {'status': 'OK', 'ansible_type': 'task', 'ansible_playbook': '/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml', 'ansible_task': 'ovirt.ovirt.hosted_engine_setup : iSCSI discover'} 2022-06-14 08:41:03,511-0600 DEBUG ansible on_any args TASK: ovirt.ovirt.hosted_engine_setup : iSCSI discover kwargs is_conditional:False 2022-06-14 08:41:03,511-0600 DEBUG ansible on_any args localhost TASK: ovirt.ovirt.hosted_engine_setup : iSCSI discover kwargs 2022-06-14 08:41:06,430-0600 DEBUG var changed: host "localhost" var "ansible_play_hosts" type "<class 'list'>" value: "[]" 2022-06-14 08:41:06,430-0600 DEBUG var changed: host "localhost" var "ansible_play_batch" type "<class 'list'>" value: "[]" 2022-06-14 08:41:06,430-0600 DEBUG var changed: host "localhost" var "play_hosts" type "<class 'list'>" value: "[]" 2022-06-14 08:41:06,431-0600 ERROR ansible failed { "ansible_host": "localhost", "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml", "ansible_result": { "_ansible_no_log": false, "changed": false, "exception": "Traceback (most recent call last):\n File \"/tmp/ansible_ovirt_host_payload_ky4zlp1s/ansible_ovirt_host_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_host.py\", line 638, in main\nTypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'\n", "invocation": { "module_args": { "activate": true, "address": null, "check_upgrade": true, "cluster": null, "comment": null, "enroll_certificate": false, "fetch_nested": false, "force": false, "hosted_engine": null, "id": null, "iscsi": { "address": "192.168.1.2", "password": "", "port": null, "username": "" }, "kdump_integration": null, "kernel_params": null, "name": "ovirt-node01.internal.net", "nested_attributes": [], "override_display": null, "override_iptables": null, "password": null, "poll_interval": 3, "power_management_enabled": null, "public_key": false, "reboot_after_installation": null, "reboot_after_upgrade": true, "spm_priority": null, "ssh_port": null, "state": "iscsidiscover", "timeout": 600, "vgpu_placement": null, "wait": true } }, "msg": "int() argument must be a string, a bytes-like object or a number, not 'NoneType'" }, "ansible_task": "iSCSI discover", "ansible_type": "task", "status": "FAILED", "task_duration": 3 } 2022-06-14 08:41:06,431-0600 DEBUG ansible on_any args <ansible.executor.task_result.TaskResult object at 0x7f9c76744370> kwargs ignore_errors:None 2022-06-14 08:41:06,432-0600 INFO ansible stats { "ansible_playbook": "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml", "ansible_playbook_duration": "00:11 Minutes", "ansible_result": "type: <class 'dict'>\nstr: {'localhost': {'ok': 5, 'failures': 1, 'unreachable': 0, 'changed': 0, 'skipped': 0, 'rescued': 0, 'ignored': 0}}", "ansible_type": "finish", "status": "FAILED" } 2022-06-14 08:41:06,432-0600 INFO SUMMARY: Duration Task Name -------- -------- [ < 1 sec ] Execute just a specific set of steps [ 00:01 ] Force facts gathering [ 00:03 ] Obtain SSO token using username/password credentials [ 00:03 ] Fetch host facts [ FAILED ] iSCSI discover 2022-06-14 08:41:06,432-0600 DEBUG ansible on_any args <ansible.executor.stats.AggregateStats object at 0x7f9c793444c0> kwargs ------ snip ------ The error suggest that the iSCSI portal port number (which is defaulted to 3260 in the UI) is not being properly passed into the python module ovirt/ovirt/plugins/modules/ovirt_host.py Looking at the code at line 638 of ovirt_host.py, found here; https://github.com/oVirt/ovirt-ansible-collection/blob/2.0.3-1/plugins/modu… I see; ..... elif state == 'iscsidiscover': host_id = get_id_by_name(hosts_service, module.params['name']) iscsi_param = module.params['iscsi'] iscsi_targets = hosts_service.service(host_id).discover_iscsi( iscsi=otypes.IscsiDetails( port=int(iscsi_param.get('port', 3260)), <---- line 638 username=iscsi_param.get('username'), password=iscsi_param.get('password'), address=iscsi_param.get('address'), portal=iscsi_param.get('portal'), ), ) ret = { 'changed': False, 'id': host_id, 'iscsi_targets': [iscsi.target for iscsi in iscsi_targets], 'iscsi_targets_struct': [get_dict_of_struct( struct=iscsi, connection=connection, fetch_nested=module.params.get('fetch_nested'), attributes=module.params.get('nested_attributes'), ) for iscsi in iscsi_targets], } ..... I'm not a Python expert, so I can't tell if this logic is correct or not. Looking at Git history on this code, it looks like the last time a change was made effecting ISCSI was in May of 2021; https://github.com/oVirt/ovirt-ansible-collection/commit/1c4c18d844a69b821f… By Martin Necas - https://github.com/mnecas I gotta believe I'm not the first one to try setting up oVirt 4.5 hosted engine with ISCSI storage. Is anyone else out there using ISCSI storage with oVirt 4.5.0.3 yet? How did you get it working?

2 2

Release announcement: go-ovirt-client 1.0.0, terraform-provider-ovirt 2.0.0
by Janos Bonic 20 Jun '22

20 Jun '22

Hey there folks, Just a quick announcement, we have released the first stable version of *go-ovirt-client*, an overlay for the go SDK which you can find here: https://github.com/ovirt/go-ovirt-client and the docs are available here: https://pkg.go.dev/github.com/ovirt/go-ovirt-client In parallel, we also released the first stable version of the *overhauled Terraform provider*, which is available on the Terraform registry: https://registry.terraform.io/providers/oVirt/ovirt/latest/docs Thank you to everyone who worked on this over the last year and made it possible. Contributors lists are here: https://github.com/oVirt/go-ovirt-client/graphs/contributors and here: https://github.com/oVirt/terraform-provider-ovirt/graphs/contributors If you run into problems, our GitHub issue box is always open. :) Cheers, Janos

1 0

Issue adding host to ovirt 4.4 cluster
by David Johnson 20 Jun '22

20 Jun '22

Good morning all, I am attempting to add a host to my ovirt 4.4 cluster. The installation of the first host went smoothly, but the installation of the second host stalled. Currently, the second host is in Installing state, but doing nothing. The installation failed due to a failure to register the host certificate. I cannot change the state of the host or retry the installation to capture logs. Warnings from the host that are visible on the ovirt console are: - Power Management is not configured for this Host. Enable Power Management - Host has no default route. - The host CPU does not match the Cluster CPU Type and is running in a degraded mode. It is missing the following CPU flags: vmx, model_Cascadelake-Server-noTSX. Please update the host CPU microcode or change the Cluster CPU Type. The error message generated at the last attempt to install the host from the ovirt console is: - Failed to enroll certificate for host ovirt-host-04 (User: admin@internal-authz). Please advise

1 1

Re: Dedicated Migration Network
by Clint Boggio 20 Jun '22

20 Jun '22

Thank you Abe, Angus, and Strahil. I appreciate your direction and input.

1 0

Re: oVirt 4.4.x step-by-step procedure to renew expired oVirt certificates
by Nathanaël Blanchet 20 Jun '22

20 Jun '22

Hi, Le 17/06/2022 à 12:18, Marko Vrgotic a écrit : > > Dear Nathanael, > > Thank you very much for you reply. Regarding host expiration playbook > you wrote – my compliments – is it safe to run on host with expired > certificates, or its rather meant to be executed for renewal of certs > on hosts with still valid certs? > both are okay, in case of a host in "up" status, it will go down during the playbook execution, but vms will continue to run without any downtime. Host will recover and go up once certificates will be successfully renewed. This is an emergency procedure, the best solution to renew a certificate on a running host is to put the host into maintenance and renew certs via UI. > We have also found following script which should at least safely take > care of the renewal of certs on host with already expired certificates > - . > > https://github.com/tothf/renew_vdsm_cert/blob/main/renew_vdsm_cert.sh > > ----- > > kind regards/met vriendelijke groeten > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > *o: *+31 (35) 6774131 > > *m: +*31 (65) 5734174** > > *e:*m.vrgotic@activevideo.com <mailto:m.vrgotic@activevideo.com> > *w: *www.activevideo.com <http://www.activevideo.com> > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein > 1.1217 WJ Hilversum, The Netherlands. The information contained in > this message may be legally privileged and confidential. It is > intended to be read only by the individual or entity to whom it is > addressed or by their designee. If the reader of this message is not > the intended recipient, you are on notice that any distribution of > this message, in any form, is strictly prohibited. If you have > received this message in error, please immediately notify the sender > and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and > delete or destroy any copy of this message. > > *From: *Nathanaël Blanchet <blanchet(a)abes.fr> > *Date: *Thursday, 16 June 2022 at 14:40 > *To: *Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org > <users(a)ovirt.org> > *Subject: *Re: [ovirt-users] oVirt 4.4.x step-by-step procedure to > renew expired oVirt certificates > > ***CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you recognize the sender!!!*** > > Hello, > > If you refer to: > > 1. engine apache certificate expiration ("PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException:) to > access to ovirt console. > => engine-setup --offline > 2. hosts certificate expiration? > => https://access.redhat.com/solutions/3532921 > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Faccess.re…> > I also wrote a playbook to do so there: > https://galaxy.ansible.com/natman/ovirt_renew_certs > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgalaxy.an…> > In this case, don't forget to renew certificate with UI (into > maintenance) when host is reponding, otherwise you may enconter > issues with console or live migration or other SSL related stuff. > > tested and approved. > > Le 16/06/2022 à 12:34, Marko Vrgotic a écrit : > > Dear oVirt, > > The oVirt SSL certificated were changed to one-year renewal and we > have a problem now. > > We are running 4.4.x version with SHE on local storage cluster and > we have four more local storage clusters. > > One the cluster running SHE, the engine and host certificates have > expired. We found the procedure for renewal prior to expiration, > but we do not have a mnual one, required once certificates have > expired. > > Would you be so kind to share the manual or steps needed to fix > our oVirt setup. > > Thank you in advance. > > ----- > > kind regards/met vriendelijke groeten > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > *o: *+31 (35) 6774131 > > *m: +*31 (65) 5734174 > > *e:*m.vrgotic@activevideo.com <mailto:m.vrgotic@activevideo.com> > *w: *www.activevideo.com <http://www.activevideo.com> > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein > 1.1217 WJ Hilversum, The Netherlands. The information contained in > this message may be legally privileged and confidential. It is > intended to be read only by the individual or entity to whom it is > addressed or by their designee. If the reader of this message is > not the intended recipient, you are on notice that any > distribution of this message, in any form, is strictly > prohibited. If you have received this message in error, please > immediately notify the sender and/or ActiveVideo Networks, LLC by > telephone at +1 408.931.9200 and delete or destroy any copy of > this message. > > > > _______________________________________________ > > Users mailing list --users(a)ovirt.org > > To unsubscribe send an email tousers-leave(a)ovirt.org > > Privacy Statement:https://www.ovirt.org/privacy-policy.html <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> > > oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> > > List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/5LOT… <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> > > -- > Nathanaël Blanchet > Supervision réseau > SIRE > 227 avenue Professeur-Jean-Louis-Viala > 34193 MONTPELLIER CEDEX 5 > Tél. 33 (0)4 67 54 84 55 > Fax 33 (0)4 67 54 84 14 > blanchet(a)abes.fr -- Nathanaël Blanchet Supervision réseau SIRE 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

1 0

After attaching the Storage domain, the VMs are disappeared from the VM import
by aminur.rahman＠iongroup.com 20 Jun '22

20 Jun '22

Hi, We're noticing some weird issue while re-attaching the storage domain. After re-attach the storage domain, some VMs are completely missing from the VM Import. Before detaching the storage domain, all the VMs were shutdown gracefully. I also noticed some disks are exists with no Alias under the disk import on the storage domain and I can't import those disks. Its failed to register the disk with <UNKONOWN> error. We're using Ovirt 4.2 with multiple Dell hosts in the cluster and Compellent SAN with iSCSI volumes. Please kindly advise if I am missing anything before detach the storage domain. Thanks

4 4

Delete the post :
by khznm 21 18 Jun '22

18 Jun '22

Dear Admin Pls delete the post under the below subject line , posted on June 18, 2022, 3:51 p.m. [ovirt-users] The he_fqdn proposed for the engine VM resolves on this host The post is incorrect and the problem was corrected sorry for the inconvenience caused. regards/khznm

1 0

The he_fqdn proposed for the engine VM resolves on this host
by khznm 21 18 Jun '22

18 Jun '22

Installing oVirt is not so easy as stated, i am struck up in FQDN resolution , i have this FQDN (vmanager.headache.com) resolve to ip 10.1.1.6 in DNS within the network,additionally i also input this entries into /etc/hosts file. while installing ( may be 8 times ) ,i am struck up with error " he_fqdn proposed for the engine VM resolves on this host " . The network is using vlan ip, the host is already installed bare metal into cisco c240-m5 with RH virtualization ver 4.4 ,its installed on IP :10.1.1.5. the problem is with installing the self hosted engine (ovirt). the vlan are like eno2.vlan7 and eno2.vlan7.1 /etc/hosts 10.1.1.6 vmanager.headache.com vmanager 10.1.1.5 vhrh1.headache.com vhrh1 10.1.1.6 vmanager.headache.com 10.1.1.5 vhrh1.headache.com i even tried cli install,,here also the same issue .. below are the error .. Host name is not valid: vmanager.headache.com resolves to 10.1.1.6 Host name is not valid: vman.headache.com did not resolve into an IP address Pls any body help here ,,,what i am doing wrong...

1 1

Cannot log into oVirt Manager - certificate issue
by Diggy Mc 18 Jun '22

18 Jun '22

I cannot log into oVirt Manager. My browser gave me a warning that the site's certificate has expired. Then when I try to log in, I receive the following error message: "PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed" How can I fix this problem? In advance, thank you for your help. hosted-engine: v4.4.8.6 hosts: oVirt Node v4.4.8.3

7 24

Re: oVirt 4.4.x step-by-step procedure to renew expired oVirt certificates
by Nathanaël Blanchet 17 Jun '22

17 Jun '22

Hello, If you refer to: 1. engine apache certificate expiration ("PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:) to access to ovirt console. => engine-setup --offline 2. hosts certificate expiration? => https://access.redhat.com/solutions/3532921 I also wrote a playbook to do so there: https://galaxy.ansible.com/natman/ovirt_renew_certs In this case, don't forget to renew certificate with UI (into maintenance) when host is reponding, otherwise you may enconter issues with console or live migration or other SSL related stuff. tested and approved. Le 16/06/2022 à 12:34, Marko Vrgotic a écrit : > > Dear oVirt, > > The oVirt SSL certificated were changed to one-year renewal and we > have a problem now. > > We are running 4.4.x version with SHE on local storage cluster and we > have four more local storage clusters. > > One the cluster running SHE, the engine and host certificates have > expired. We found the procedure for renewal prior to expiration, but > we do not have a mnual one, required once certificates have expired. > > Would you be so kind to share the manual or steps needed to fix our > oVirt setup. > > Thank you in advance. > > ----- > > kind regards/met vriendelijke groeten > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > *o: *+31 (35) 6774131 > > *m: +*31 (65) 5734174** > > *e:*m.vrgotic@activevideo.com <mailto:m.vrgotic@activevideo.com> > *w: *www.activevideo.com <http://www.activevideo.com> > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein > 1.1217 WJ Hilversum, The Netherlands. The information contained in > this message may be legally privileged and confidential. It is > intended to be read only by the individual or entity to whom it is > addressed or by their designee. If the reader of this message is not > the intended recipient, you are on notice that any distribution of > this message, in any form, is strictly prohibited. If you have > received this message in error, please immediately notify the sender > and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and > delete or destroy any copy of this message. > > > _______________________________________________ > Users mailing list --users(a)ovirt.org > To unsubscribe send an email tousers-leave(a)ovirt.org > Privacy Statement:https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ > List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/5LOT… -- Nathanaël Blanchet Supervision réseau SIRE 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

3 4

Cant fix network
by David Johnson 17 Jun '22

17 Jun '22

Good afternoon all, Welcome to the third part of my trilogy of disaster recovery woes. Much of my woe is self inflicted, but I have learned enough from this that I now know how to ask the right question about the original problem that got me here. After reinstalling the engine from the ground up, ovirt immediately recognized the first host (yay) whose network configuration I had manually fixed on the host console. I removed the host from the engine, stripped its system down to the bare OS, then attempted to reinstall ovirt. Apparently the original network configuration is still saved on the host, because reinstall kicks the host off of the management network, replacing the good configuration that I just made on the host with the bad one that was the original cause of my heartache. What is wrong with this configuration is that the storage network is on a 10 gbit sfp+ physical network, isolated from the 1 gbit ovirtmgt network with RJ12 connectors. There is no way to bridge the two networks Here is the original (bad) configuration: [image: image.png] Here is the corrected configuration. Note that it wont let me connect to the network. it's frustrating because I know that the engine is not communicating with the host, and I can map out the fix here, but I can't save it. [image: image.png] It will not allow me to remove the host or switch the host to inoperable or maintenance mode. Please advise.

2 2

1 Unsynced Entry
by Abe E 16 Jun '22

16 Jun '22

I am having an issue on 1 single node where it is reporting 1 unsynced entry and it is not healing by itself or by force. In the glustershd.log it is reporting the following: W [MSGID: 114031] [client-rpc-fops_v2.c:2620:client4_0_lookup_cbk] 0-data-client-2: remote operation failed. [{path=<gfid:a8769008-3658-400f-b3ba-c0df40e648a7>}, {gfid=a8769008-3658-400f-b3ba-c0df40e648a7}, {errno=2}, {error=No such file or directory}] I assume if it was missing it would look to other nodes for it but I understand from this that even other nodes do not have it?

1 0

OVIRT Package Upgrade Interrupted
by Abe E 16 Jun '22

16 Jun '22

It seems I haven't learned my lesson with upgrading via GUI. For some reason my server crashed during use of the GUI Upgrade method and it looks like maybe it did not complete or the like. 3NodeGluster I then noticed that I had actually upgrading a single node to the 4.5.1 Testing Version, that being said my glusterfs for this node is on 10.2 while other nodes are on 10.1. Is there a "cleaner" way of reverting the upgrade? I assume there are other packages that may have been upgraded that should be reverted.

2 4

Dead agent
by Valerio Luccio 16 Jun '22

16 Jun '22

Hello all, I have an ovirt 4.4 installation whit self-hosted engine where the agent seems to have died. The VMs are still running, so I assume that the engine itself is still running (is this a wrong assumption ?). Can I restart the agent without affecting the running VMs, that is how will restarting the agent affect the running VMs ? If I can restart the agent, what's the correct way of doing it ? Thanks, -- Valerio Luccio High Performance Computing 10 Astor Place, Room 416D New York University New York, NY 10003 "In an open world, who needs windows or gates ?"

2 2

oVirt 4.4.x step-by-step procedure to renew expired oVirt certificates
by Marko Vrgotic 16 Jun '22

16 Jun '22

Dear oVirt, The oVirt SSL certificated were changed to one-year renewal and we have a problem now. We are running 4.4.x version with SHE on local storage cluster and we have four more local storage clusters. One the cluster running SHE, the engine and host certificates have expired. We found the procedure for renewal prior to expiration, but we do not have a mnual one, required once certificates have expired. Would you be so kind to share the manual or steps needed to fix our oVirt setup. Thank you in advance. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

libvirtd: dependencies is missing or not an array
by jb 16 Jun '22

16 Jun '22

Hello everybody, in the Protocols from node cockpit, I get some times a day this error: internal error: dependencies is missing or not an array CODE_FILE ../src/util/virjson.c CODE_FUNC virJSONValueObjectGetStringArray CODE_LINE 1324 LIBVIRT_CODE 1 LIBVIRT_DOMAIN 0 LIBVIRT_SOURCE util.error PRIORITY 3 SYSLOG_FACILITY 24 _BOOT_ID 0d6bdd9755d44dac8afe2207f6d69f8d _CAP_EFFECTIVE 1ffffffffff _CMDLINE /usr/sbin/libvirtd _COMM libvirtd _EXE /usr/sbin/libvirtd _GID 0 _HOSTNAME onode1.example.org _MACHINE_ID 977a6d184c604d42ad81ac03904ccba7 _PID 2505 _SELINUX_CONTEXT system_u:system_r:virtd_t:s0-s0:c0.c1023 _SOURCE_REALTIME_TIMESTAMP 1655362628796631 _SYSTEMD_CGROUP /system.slice/libvirtd.service _SYSTEMD_INVOCATION_ID f7685b24f045456aac53ce1673f143c7 _SYSTEMD_SLICE system.slice _SYSTEMD_UNIT libvirtd.service _TRANSPORT journal Is this something I have to worry about it?

1 0

Ovirt manager engine certificate expired
by Don Dupuis 16 Jun '22

16 Jun '22

Hello I have an older installation that was running just great and the certificate expired, it is version 4.2.8.2. I ran the engine-setup as per this "access.redhat.com/solutions/4780411". It didn't update the certificate or the dates and stil have the same issue. Any ideas to fix this issue? Thanks Don

1 2

oVirt 4.5 linux guest vm with host device added to it fails to start
by Don Dupuis 15 Jun '22

15 Jun '22

Hello I have a RHEL 8.6 based hypervisor with a Mellanox ConnectX-5 IB card installed with SRIOV enabled. The host device I am assigning is pci_0000_af_00_2. The card is working as I can talk to other infiniband interfaces on other servers. Below is the output of lspci. 3b:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 3b:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5] af:00.1 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.2 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.3 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] af:00.4 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] The linux vm is configured as Q35 Chipset with UEFI, 16 cpus, numa enabled, and cpu pinning enabled. OS is RHEL 7.9. As soon as I start the vm, I get an immediate error message stating "Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable." If I remove the host device from the vm config, then it starts and runs fine. This setup was working just fine on RHEL8.4 and oVirt 4.4.7 using the proper driver for RHEL 8.4. Here is the engine.log after I press the run button. 2022-06-10 11:22:10,506-05 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] Lock Acquired to object 'EngineLock:{exclusiveLocks='[de54b903-7204-4966-95a3-05f64ed17f68=VM]', sharedLocks=''}' 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='de54b903-7204-4966-95a3-05f64ed17f68'}), log id: 6faf22a5 2022-06-10 11:22:10,520-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-1) [81144b66-e5f9-474e-a922-e2ce49cdc8ca] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6faf22a5 2022-06-10 11:22:10,560-05 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (default task-1) [] Candidate host 'rvsh002' ('f68352c2-6ddc-44ae-a19b-9262e92327f8') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'HostDevice' (correlation id: null) 2022-06-10 11:22:10,569-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM ws006 due to a failed validation: [Cannot run VM. There is no host that satisfies current scheduling constraints. See below for details:, The host rvsh002 did not satisfy internal filter HostDevice because some of the required host devices are unavailable.] (User: admin@internal-authz). 2022-06-10 11:22:10,569-05 WARN [org.ovirt.engine.core.bll.RunVmCommand] (default task-1) [] Validation of action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName rvsh002,$filterName HostDevice,VAR__DETAIL__HOST_DEVICE_UNAVAILABLE,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL 2022-06-10 11:22:10,570-0 There was nothing in the vdsm.log on the hypervisor related to this issue that I could see after hitting the run button. Thanks Don

2 4

dnf update fails on ovirt node 4.5.0-2022052513
by pat＠patfruth.com 14 Jun '22

14 Jun '22

I have freshly install ovirt node 4.5 from the iso download here; https://resources.ovirt.org/pub/ovirt-4.5/iso/ovirt-node-ng-installer/4.5.0… The installation appears to have been successful. Now, when I attempt apply the latest updates with 'dnf update', I get an error [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 Here is the entire output of the dnf update command; Last metadata expiration check: 0:47:08 ago on Mon 13 Jun 2022 10:55:35 PM MDT. Dependencies resolved. ======================================================================================================================================================================= Package Architecture Version Repository Size ======================================================================================================================================================================= Installing: ovirt-node-ng-image-update noarch 4.5.1-0.1.el8 ovirt-45-upstream-testing 1.1 G replacing ovirt-node-ng-image-update-placeholder.noarch 4.5.0.3-1.el8 Transaction Summary ======================================================================================================================================================================= Install 1 Package Total download size: 1.1 G Is this ok [y/N]: y Downloading Packages: [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [MIRROR] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: Interrupted by header callback: Server reports Content-Length: 1180431893 but expected size is: 1180857942 [FAILED] ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch.rpm: No more mirrors to try - All mirrors were already tried without success The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: Error downloading packages: ovirt-node-ng-image-update-4.5.1-0.1.el8.noarch: Cannot download, all mirrors were already tried without success How can I fix this error, and get the latest updates for 4.5 installed?

5 13

Initramfs and vmlinuz corrupted, how to recover?
by douglasddr8＠gmail.com 14 Jun '22

14 Jun '22

I have a Dell server with a BIOS bug, it corrupted the files during a UEFI boot. I noticed that the initramfs and vmlinuz files are zero-sized How can I retrieve or generate new ones?

3 2

4.4.10 -> 4.5 upgrade on gluster based setups
by Gilboa Davara 14 Jun '22

14 Jun '22

Hello all, I've more or less finished upgrading all my "small" oVirt setups (single hosts, multiple hosts + NFS, etc) and I'm now ready to start upgrading my main production setups. (All using 3 x nodes / gluster setup). A couple of questions: 1. Should I use the normal sequence, as in: engine, hosts, hosts, hosts? 2. Was the gluster bug below still open in the latest oVirt 4.5 release? https://lists.ovirt.org/archives/list/users@ovirt.org/thread/4LGBUOEBV7YNES… Thanks, Gilboa

2 2

Reinstall standalone node without vms loss
by douglasddr8＠gmail.com 14 Jun '22

14 Jun '22

My server failed and I can't boot via UEFI How can I reinstall this node (standalone) without losing my virtual machines?. I checked the filesystem and it's completely intact, I couldn't figure out what caused the UEFI to fail.

2 1

Install Ovirt 4.4.10 to standalone system from iso fails
by David Johnson 14 Jun '22

14 Jun '22

Hi everyone, I am attempting to reinstall as a clean system, and the install is failing silently. At the point where I should be able to run engine-setup, the engine setup utility is absent. Installation medium: iso downloaded from https://resources.ovirt.org/pub/ovirt-4.4/iso/ovirt-node-ng-installer/4.4.1… As I attempt to install the missing dependencies, I keep getting deeper into missing rpm land. If this iso is no longer any good, even though it is only 4 months old, it should probably be documented as bad or art least superceded by a fixed iso.

3 12

can't use vmconsole anymore
by Nathanaël Blanchet 13 Jun '22

13 Jun '22

Hi, I was used to use the vmconsole proxy, but since a while, I'm getting this issue (currently 4.4.5): # ssh -t -p 2222 ovirt-vmconsole(a)air.v100.abes.fr connect ovirt-vmconsole(a)air.v100.abes.fr: Permission denied (publickey). I found following in the engine.log 2021-04-15 17:55:43,094+02 ERROR [org.ovirt.engine.core.services.VMConsoleProxyServlet] (default task-4) [] Error validating ticket: : sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at org.ovirt.engine.core.uutils//org.ovirt.engine.core.uutils.crypto.CertificateChain.buildCertPath(CertificateChain.java:128) at org.ovirt.engine.core.uutils//org.ovirt.engine.core.uutils.crypto.ticket.TicketDecoder.decode(TicketDecoder.java:89) at deployment.engine.ear.services.war//org.ovirt.engine.core.services.VMConsoleProxyServlet.validateTicket(VMConsoleProxyServlet.java:175) at deployment.engine.ear.services.war//org.ovirt.engine.core.services.VMConsoleProxyServlet.doPost(VMConsoleProxyServlet.java:225) The user key is the good one, I use the same with my other engines and I can successfully connect to vm consoles. Thank you for helping -- Nathanaël Blanchet Supervision réseau SIRE 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

5 16

Host fails to activate
by David Johnson 12 Jun '22

12 Jun '22

Good afternoon all, Ovirt version: 4.14.4.10.7-1.el8 Centos version: Linux version 4.18.0-365.el8.x86_64 ( mockbuild(a)kbuilder.bsys.centos.org) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-10) (GCC)) #1 SMP Thu Feb 10 16:11:23 UTC 2022 Background: We had a mother board fail in our storage device. I was able to migrate the storage domain to the backup device before it failed completely, and have been running on the backup device for several weeks while we purchased a replacement main storage. Today I shut everything down cleanly, replaced the main storage, and restarted the cluster. We did disconnect and reconnect the network on all of the devices as we shuffled equipment in the rack. One of the hosts in the cluster refuses to come back up.I am able to connect to the host via putty. Ovirt gui reporting: Setting Host ovirt-host-03.maxisinc.net to Non-Operational mode. Completed: Jun 11, 2022, 4:59:57 PM Activating Host ovirt-host-03.maxisinc.net Completed: Jun 11, 2022, 4:59:57 PM Invoking Activate Host ovirt-host-03.maxisinc.net Completed: Jun 11, 2022, 4:57:40 PM Installing Host ovirt-host-03.maxisinc.net log from host is 5:09 PM GetManagedObjects() failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. pulseaudio 4:55 PM bondscan-DGwC1l: option lacp_active: mode dependency failed, not supported in mode balance-alb(6) kernel 4:55 PM bondscan-DGwC1l: option arp_all_targets: invalid value (2) kernel 4:55 PM bondscan-DGwC1l: option fail_over_mac: invalid value (3) kernel 4:55 PM bondscan-DGwC1l: option primary_reselect: invalid value (3) kernel 4:55 PM bondscan-DGwC1l: option ad_select: invalid value (3) kernel 4:55 PM

2 6

oVirt SSH rate limit and disable SSH passwd auth
by tasnadi.peter＠kifu.gov.hu 10 Jun '22

10 Jun '22

Hello, 1. Is it possible to disable ssh root password authentication in a working oVirt cluster without any problems? (host and ovirt-engine) /etc/ssh/sshd_config PasswordAuthentication no The SSH public authentication key is set on the host. 2. I tried setting ssh rate limit using firewall-cmd but it doesn't work for some reason. I can log in more than once. firewall-cmd --permanent --add-rich-rule='rule family="ipv4" priority="-1" service name=ssh limit value=3/m accept' There is best practice for this? Thanks Peter

2 1

[ANN] oVirt 4.5.1 First Release Candidate is now available for testing
by Lev Veyde 09 Jun '22

09 Jun '22

oVirt 4.5.1 First Release Candidate is now available for testing The oVirt Project is pleased to announce the availability of oVirt 4.5.1 First Release Candidate for testing, as of June 9th, 2022. This update is the first in a series of stabilization updates to the 4.5 series. Documentation - If you want to try oVirt as quickly as possible, follow the instructions on the Download <https://ovirt.org/download/> page. - For complete installation, administration, and usage instructions, see the oVirt Documentation <https://ovirt.org/documentation/>. - For upgrading from a previous version, see the oVirt Upgrade Guide <https://ovirt.org/documentation/upgrade_guide/>. - For a general overview of oVirt, see About oVirt <https://ovirt.org/community/about.html>. Important notes before you try it Please note this is a pre-release build. The oVirt Project makes no guarantees as to its suitability or usefulness. This pre-release must not be used in production. Installation instructions For installation instructions and additional information please refer to: https://ovirt.org/documentation/ This release is available now on x86_64 architecture for: - CentOS Stream 8 - RHEL 8.6 Beta and derivatives This release supports Hypervisor Hosts on x86_64: - oVirt Node NG (based on CentOS Stream 8) - CentOS Stream 8 - RHEL 8.6 Beta and derivatives Builds are also available for ppc64le and aarch64. Experimental builds for CentOS Stream 9 are also provided for Hypervisor Hosts. See the release notes [1] for installation instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available based on CentOS Stream 8 - oVirt Node NG is already available based on CentOS Stream 8 Additional Resources: * Read more about the oVirt 4.5.1 pre-release highlights: http://www.ovirt.org/release/4.5.1/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.5.1/ [2] http://resources.ovirt.org/pub/ovirt-4.5-pre/iso/ Thanks in advance, -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

1 0

VmMediatedDevices help for vGPU in overt 4.5
by Don Dupuis 09 Jun '22

09 Jun '22

Hello I am looking for an example on how to use the new VmMediatedDevices service to add an Nvidia vGPU in ovirt to guest vms. I had it working just fine in oVirt 4.4 when using the custom_properties method. Just need to understand the new method/way of doing it correctly using the python3-ovirt-engine-sdk. Thanks Don

2 2

Re: list-view instead of tiled-view in oVirt VM Portal?
by Frank Coons 09 Jun '22

09 Jun '22

Please note that (a) there are people that use more than 20 VM's that do not need admin access, and (b) some people do not LIKE looking at big gaudy buttons, even if there are only 15 of them. I put in an RFE to bring back the list view YEARS ago and was basically told that "we know what you want better than you do." I am willing to bet that many more people want the list view than you realize, but you don't seem to be willing to listen. Disgruntled.

2 1

ovirt 4.4.10 ansible version
by Kapetanakis Giannis 09 Jun '22

09 Jun '22

Could someone verify the correct ansible version for ovirt 4.4.10 ? I'm having dependencies problem: # rpm -q ansible ansible-2.9.27-3.el8.noarch # dnf update Last metadata expiration check: 1:14:38 ago on Thu 09 Jun 2022 10:59:53 EEST. Error: Problem: package ovirt-engine-4.4.10.7-1.el8.noarch requires ansible < 2.10.0, but none of the providers can be installed - cannot install both ansible-5.4.0-2.el8.noarch and ansible-2.9.27-3.el8.noarch - cannot install both ansible-2.9.17-1.el8.noarch and ansible-5.4.0-2.el8.noarch - cannot install both ansible-2.9.18-2.el8.noarch and ansible-5.4.0-2.el8.noarch - cannot install both ansible-2.9.20-2.el8.noarch and ansible-5.4.0-2.el8.noarch - cannot install both ansible-2.9.21-2.el8.noarch and ansible-5.4.0-2.el8.noarch - cannot install both ansible-2.9.23-2.el8.noarch and ansible-5.4.0-2.el8.noarch - cannot install both ansible-2.9.24-2.el8.noarch and ansible-5.4.0-2.el8.noarch - cannot install both ansible-2.9.27-2.el8.noarch and ansible-5.4.0-2.el8.noarch - cannot install the best update candidate for package ovirt-engine-4.4.10.7-1.el8.noarch - cannot install the best update candidate for package ansible-2.9.27-3.el8.noarch - package ansible-2.9.20-1.el8.noarch is filtered out by exclude filtering - package ansible-2.9.16-1.el8.noarch is filtered out by exclude filtering - package ansible-2.9.19-1.el8.noarch is filtered out by exclude filtering - package ansible-2.9.23-1.el8.noarch is filtered out by exclude filtering (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) # dnf list --showduplicates ansible Last metadata expiration check: 1:15:11 ago on Thu 09 Jun 2022 10:59:53 EEST. Installed Packages ansible.noarch 2.9.27-3.el8 @epel Available Packages ansible.noarch 2.9.17-1.el8 ovirt-4.4-centos-ovirt44 ansible.noarch 2.9.18-2.el8 ovirt-4.4-centos-ovirt44 ansible.noarch 2.9.20-2.el8 ovirt-4.4-centos-ovirt44 ansible.noarch 2.9.21-2.el8 ovirt-4.4-centos-ovirt44 ansible.noarch 2.9.23-2.el8 ovirt-4.4-centos-ovirt44 ansible.noarch 2.9.24-2.el8 ovirt-4.4-centos-ovirt44 ansible.noarch 2.9.27-2.el8 ovirt-4.4-centos-ovirt44 ansible.noarch 5.4.0-2.el8 epel ansible.noarch 5.4.0-2.el8 ovirt-4.4-epel thanks, G

2 1

Re: Self-hosted engine failing liveliness check
by McNamara, Bradley 09 Jun '22

09 Jun '22

When a run "hosted-engine --check-liveliness" it returns "Hosted Engine is not up!" When I run this command and can see it hitting the httpd server, with success, in the httpd logs. Accessing the URL returns this: "DB Up!Welcome to Health Status!". ________________________________ From: McNamara, Bradley <Bradley.McNamara(a)seattle.gov> Sent: Wednesday, June 8, 2022 12:54 PM To: users(a)ovirt.org <users(a)ovirt.org> Subject: [ovirt-users] Self-hosted engine failing liveliness check CAUTION: External Email Hello, and thank you all for your help. I'm running Oracle's rebranded oVirt 4.3.10. All has been good until I patched my self-hosted engine. I ran through the normal process: backup, global maintenance mode, update the oVirt packages, run engine-setup, etc. All completed normally without issues. I rebooted the self-hosted engine VM, and now it constantly fails liveliness checks and the HA agent reboots it every five minutes, or so. I put it in back in global maintenance so the HA agent would not reboot it. The VM is up and works correctly. I can do everything normally. From what I can tell the HA agent liveliness check is just a http get to the web portal. I can see that happening with success. What is the lilveliness check actually doing? All services on the VM are up and running without issue. Where can I look to figure this out? Here is the output of hosted-engine --vm-status: [root@itdlolv101 ~]# hosted-engine --vm-status !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host itdlolv100.ci.seattle.wa.us (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : itdlolv100.ci.seattle.wa.us Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 855e161f local_conf_timestamp : 55128 Host timestamp : 55128 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=55128 (Wed Jun 8 12:52:20 2022) host-id=1 score=3400 vm_conf_refresh_time=55128 (Wed Jun 8 12:52:20 2022) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False --== Host itdlolv101.ci.seattle.wa.us (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : itdlolv101.ci.seattle.wa.us Host ID : 2 Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : cc1c2261 local_conf_timestamp : 45453 Host timestamp : 45453 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=45453 (Wed Jun 8 12:55:15 2022) host-id=2 score=3400 vm_conf_refresh_time=45453 (Wed Jun 8 12:55:15 2022) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! [root@itdlolv101 ~]#

1 0

Re: storage high latency, sanlock errors, cluster instability
by Nir Soffer 08 Jun '22

08 Jun '22

On Sun, May 29, 2022 at 9:03 PM Jonathan Baecker <jonbae77(a)gmail.com> wrote: > > Am 29.05.22 um 19:24 schrieb Nir Soffer: > > On Sun, May 29, 2022 at 7:50 PM Jonathan Baecker <jonbae77(a)gmail.com> wrote: > > Hello everybody, > > we run a 3 node self hosted cluster with GlusterFS. I had a lot of problem upgrading ovirt from 4.4.10 to 4.5.0.2 and now we have cluster instability. > > First I will write down the problems I had with upgrading, so you get a bigger picture: > > engine update when fine > But nodes I could not update because of wrong version of imgbase, so I did a manual update to 4.5.0.1 and later to 4.5.0.2. First time after updating it was still booting into 4.4.10, so I did a reinstall. > Then after second reboot I ended up in the emergency mode. After a long searching I figure out that lvm.conf using use_devicesfile now but there it uses the wrong filters. So I comment out this and add the old filters back. This procedure I have done on all 3 nodes. > > When use_devicesfile (default in 4.5) is enabled, lvm filter is not > used. During installation > the old lvm filter is removed. > > Can you share more info on why it does not work for you? > > The problem was, that the node could not mount the gluster volumes anymore and ended up in emergency mode. > > - output of lsblk > > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda 8:0 0 1.8T 0 disk > `-XA1920LE10063_HKS028AV 253:0 0 1.8T 0 mpath > |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda_tmeta 253:16 0 9G 0 lvm > | `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda-tpool 253:18 0 1.7T 0 lvm > | |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda 253:19 0 1.7T 1 lvm > | |-gluster_vg_sda-gluster_lv_data 253:20 0 100G 0 lvm /gluster_bricks/data > | `-gluster_vg_sda-gluster_lv_vmstore 253:21 0 1.6T 0 lvm /gluster_bricks/vmstore > `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda_tdata 253:17 0 1.7T 0 lvm > `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda-tpool 253:18 0 1.7T 0 lvm > |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda 253:19 0 1.7T 1 lvm > |-gluster_vg_sda-gluster_lv_data 253:20 0 100G 0 lvm /gluster_bricks/data > `-gluster_vg_sda-gluster_lv_vmstore 253:21 0 1.6T 0 lvm /gluster_bricks/vmstore > sr0 11:0 1 1024M 0 rom > nvme0n1 259:0 0 238.5G 0 disk > |-nvme0n1p1 259:1 0 1G 0 part /boot > |-nvme0n1p2 259:2 0 134G 0 part > | |-onn-pool00_tmeta 253:1 0 1G 0 lvm > | | `-onn-pool00-tpool 253:3 0 87G 0 lvm > | | |-onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1 253:4 0 50G 0 lvm / > | | |-onn-pool00 253:7 0 87G 1 lvm > | | |-onn-home 253:8 0 1G 0 lvm /home > | | |-onn-tmp 253:9 0 1G 0 lvm /tmp > | | |-onn-var 253:10 0 15G 0 lvm /var > | | |-onn-var_crash 253:11 0 10G 0 lvm /var/crash > | | |-onn-var_log 253:12 0 8G 0 lvm /var/log > | | |-onn-var_log_audit 253:13 0 2G 0 lvm /var/log/audit > | | |-onn-ovirt--node--ng--4.5.0.1--0.20220511.0+1 253:14 0 50G 0 lvm > | | `-onn-var_tmp 253:15 0 10G 0 lvm /var/tmp > | |-onn-pool00_tdata 253:2 0 87G 0 lvm > | | `-onn-pool00-tpool 253:3 0 87G 0 lvm > | | |-onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1 253:4 0 50G 0 lvm / > | | |-onn-pool00 253:7 0 87G 1 lvm > | | |-onn-home 253:8 0 1G 0 lvm /home > | | |-onn-tmp 253:9 0 1G 0 lvm /tmp > | | |-onn-var 253:10 0 15G 0 lvm /var > | | |-onn-var_crash 253:11 0 10G 0 lvm /var/crash > | | |-onn-var_log 253:12 0 8G 0 lvm /var/log > | | |-onn-var_log_audit 253:13 0 2G 0 lvm /var/log/audit > | | |-onn-ovirt--node--ng--4.5.0.1--0.20220511.0+1 253:14 0 50G 0 lvm > | | `-onn-var_tmp 253:15 0 10G 0 lvm /var/tmp > | `-onn-swap 253:5 0 20G 0 lvm [SWAP] > `-nvme0n1p3 259:3 0 95G 0 part > `-gluster_vg_nvme0n1p3-gluster_lv_engine 253:6 0 94G 0 lvm /gluster_bricks/engine > > - The old lvm filter used, and why it was needed > > filter = ["a|^/dev/disk/by-id/lvm-pv-uuid-Nn7tZl-TFdY-BujO-VZG5-EaGW-5YFd-Lo5pwa$|", "a|^/dev/disk/by-id/lvm-pv-uuid-Wcbxnx-2RhC-s1Re-s148-nLj9-Tr3f-jj4VvE$|", "a|^/dev/disk/by-id/lvm-pv-uuid-lX51wm-H7V4-3CTn-qYob-Rkpx-Tptd-t94jNL$|", "r|.*|"] > > I don't remember exactly any more why it was needed, but without the node was not working correctly. I think I even used vdsm-tool config-lvm-filter. I think that if you list the devices in this filter: ls -lh /dev/disk/by-id/lvm-pv-uuid-Nn7tZl-TFdY-BujO-VZG5-EaGW-5YFd-Lo5pwa \ /dev/disk/by-id/lvm-pv-uuid-Wcbxnx-2RhC-s1Re-s148-nLj9-Tr3f-jj4VvE \ /dev/disk/by-id/lvm-pv-uuid-lX51wm-H7V4-3CTn-qYob-Rkpx-Tptd-t94jNL You will see that these are the devices used by these vgs: gluster_vg_sda, gluster_vg_nvme0n1p3, onn > > - output of vdsm-tool config-lvm-filter > > Analyzing host... > Found these mounted logical volumes on this host: > > logical volume: /dev/mapper/gluster_vg_nvme0n1p3-gluster_lv_engine > mountpoint: /gluster_bricks/engine > devices: /dev/nvme0n1p3 > > logical volume: /dev/mapper/gluster_vg_sda-gluster_lv_data > mountpoint: /gluster_bricks/data > devices: /dev/mapper/XA1920LE10063_HKS028AV > > logical volume: /dev/mapper/gluster_vg_sda-gluster_lv_vmstore > mountpoint: /gluster_bricks/vmstore > devices: /dev/mapper/XA1920LE10063_HKS028AV > > logical volume: /dev/mapper/onn-home > mountpoint: /home > devices: /dev/nvme0n1p2 > > logical volume: /dev/mapper/onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1 > mountpoint: / > devices: /dev/nvme0n1p2 > > logical volume: /dev/mapper/onn-swap > mountpoint: [SWAP] > devices: /dev/nvme0n1p2 > > logical volume: /dev/mapper/onn-tmp > mountpoint: /tmp > devices: /dev/nvme0n1p2 > > logical volume: /dev/mapper/onn-var > mountpoint: /var > devices: /dev/nvme0n1p2 > > logical volume: /dev/mapper/onn-var_crash > mountpoint: /var/crash > devices: /dev/nvme0n1p2 > > logical volume: /dev/mapper/onn-var_log > mountpoint: /var/log > devices: /dev/nvme0n1p2 > > logical volume: /dev/mapper/onn-var_log_audit > mountpoint: /var/log/audit > devices: /dev/nvme0n1p2 > > logical volume: /dev/mapper/onn-var_tmp > mountpoint: /var/tmp > devices: /dev/nvme0n1p2 > > Configuring LVM system.devices. > Devices for following VGs will be imported: > > gluster_vg_sda, gluster_vg_nvme0n1p3, onn > > To properly configure the host, we need to add multipath > blacklist in /etc/multipath/conf.d/vdsm_blacklist.conf: > > blacklist { > wwid "eui.0025388901b1e26f" > } > > > Configure host? [yes,NO] If you run "vdsm-tool config-lvm-filter" and confirm with "yes", I think all the vgs will be imported properly into lvm devices file. I don't think it will solve the storage issues you have since Feb 2022, but at least you will have a standard configuration and the next upgrade will not revert your local settings. > If using lvm devices does not work for you, you can enable the lvm > filter in vdsm configuration > by adding a drop-in file: > > $ cat /etc/vdsm/vdsm.conf.d/99-local.conf > [lvm] > config_method = filter > > And run: > > vdsm-tool config-lvm-filter > > to configure the lvm filter in the best way for vdsm. If this does not create > the right filter we would like to know why, but in general you should use > lvm devices since it avoids the trouble of maintaining the filter and dealing > with upgrades and user edited lvm filter. > > If you disable use_devicesfile, the next vdsm upgrade will enable it > back unless > you change the configuration. > > I would be happy to just use the default, when there is a way to make use_devicesfile to wok. > > Also even if you disable use_devicesfile in lvm.conf, vdsm still use > --devices instead > of filter when running lvm commands, and lvm commands run by vdsm ignore your > lvm filter since the --devices option overrides the system settings. > > ... > > I notice some unsync volume warning, but because I had this in the past to, after upgrading, I though after some time they will disappear. The next day there still where there, so I decided to put the nodes again in the maintenance mode and restart the glusterd service. After some time the sync warnings where gone. > > Not clear what these warnings are, I guess Gluster warning? > > Yes was Gluster warnings under Storage -> Volumes it was saying that some entries are unsync. > > So now the actual problem: > > Since this time the cluster is unstable. I get different errors and warning, like: > > VM [name] is not responding > out of nothing HA VM gets migrated > VM migration can fail > VM backup with snapshoting and export take very long > > How do you backup the vms? do you sue a backup application? how is it > configured? > > I use a self made plython script, which uses the rest api. I create a snapshot from the VM, build a new VM from that snapshot and move the new one to the export domain. This is not very efficient - this copy the entire vm at the point of time of the snapshot and then copy it again to the export domain. If you use a backup application supporting the incremental backup API, the first full backup will copy the entire vm once, but later incremental backup will copy only the changes since the last backup. > > VMs are getting very slow some times > Storage domain vmstore experienced a high latency of 9.14251 > ovs|00001|db_ctl_base|ERR|no key "dpdk-init" in Open_vSwitch record "." column other_config > 489279 [1064359]: s8 renewal error -202 delta_length 10 last_success 489249 > 444853 [2243175]: s27 delta_renew read timeout 10 sec offset 0 /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/ids > 471099 [2243175]: s27 delta_renew read timeout 10 sec offset 0 /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/ids > many of: 424035 [2243175]: s27 delta_renew long write time XX sec > > All these issues tell use that your storage is not working correctly. > > sanlock.log is full of renewal errors form May: > > $ grep 2022-05- sanlock.log | wc -l > 4844 > > $ grep 2022-05- sanlock.log | grep 'renewal error' | wc -l > 631 > > But there is lot of trouble from earlier months: > > $ grep 2022-04- sanlock.log | wc -l > 844 > $ grep 2022-04- sanlock.log | grep 'renewal error' | wc -l > 29 > > $ grep 2022-03- sanlock.log | wc -l > 1609 > $ grep 2022-03- sanlock.log | grep 'renewal error' | wc -l > 483 > > $ grep 2022-02- sanlock.log | wc -l > 826 > $ grep 2022-02- sanlock.log | grep 'renewal error' | wc -l > 242 > > Here sanlock log looks healthy: > > $ grep 2022-01- sanlock.log | wc -l > 3 > $ grep 2022-01- sanlock.log | grep 'renewal error' | wc -l > 0 > > $ grep 2021-12- sanlock.log | wc -l > 48 > $ grep 2021-12- sanlock.log | grep 'renewal error' | wc -l > 0 > > vdsm log shows that 2 domains are not accessible: > > $ grep ERROR vdsm.log > 2022-05-29 15:07:19,048+0200 ERROR (check/loop) [storage.monitor] > Error checking path > /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata > (monitor:511) > 2022-05-29 16:33:59,049+0200 ERROR (check/loop) [storage.monitor] > Error checking path > /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata > (monitor:511) > 2022-05-29 16:34:39,049+0200 ERROR (check/loop) [storage.monitor] > Error checking path > /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata > (monitor:511) > 2022-05-29 17:21:39,050+0200 ERROR (check/loop) [storage.monitor] > Error checking path > /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata > (monitor:511) > 2022-05-29 17:55:59,712+0200 ERROR (check/loop) [storage.monitor] > Error checking path > /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata > (monitor:511) > 2022-05-29 17:56:19,711+0200 ERROR (check/loop) [storage.monitor] > Error checking path > /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata > (monitor:511) > 2022-05-29 17:56:39,050+0200 ERROR (check/loop) [storage.monitor] > Error checking path > /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata > (monitor:511) > 2022-05-29 17:56:39,711+0200 ERROR (check/loop) [storage.monitor] > Error checking path > /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata > (monitor:511) > > You need to find what is the issue with your Gluster storage. > > I hope that Ritesh can help debug the issue with Gluster. > > Nir > > I'm worry that I do something, that it makes it even more worst, and I hove not idea what's the problem. To me it looks not exactly like a problem with data inconsistencies. The problem is that your Gluster storage is not healthy, and reading and writing to it times out. Please keep users(a)ovirt.org CC when you reply. Gluster storage is very popular in this mailing list and you may get useful help from other users. Nir

4 3

Self-hosted engine failing liveliness check
by McNamara, Bradley 08 Jun '22

08 Jun '22

Hello, and thank you all for your help. I'm running Oracle's rebranded oVirt 4.3.10. All has been good until I patched my self-hosted engine. I ran through the normal process: backup, global maintenance mode, update the oVirt packages, run engine-setup, etc. All completed normally without issues. I rebooted the self-hosted engine VM, and now it constantly fails liveliness checks and the HA agent reboots it every five minutes, or so. I put it in back in global maintenance so the HA agent would not reboot it. The VM is up and works correctly. I can do everything normally. From what I can tell the HA agent liveliness check is just a http get to the web portal. I can see that happening with success. What is the lilveliness check actually doing? All services on the VM are up and running without issue. Where can I look to figure this out? Here is the output of hosted-engine --vm-status: [root@itdlolv101 ~]# hosted-engine --vm-status !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host itdlolv100.ci.seattle.wa.us (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : itdlolv100.ci.seattle.wa.us Host ID : 1 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} Score : 3400 stopped : False Local maintenance : False crc32 : 855e161f local_conf_timestamp : 55128 Host timestamp : 55128 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=55128 (Wed Jun 8 12:52:20 2022) host-id=1 score=3400 vm_conf_refresh_time=55128 (Wed Jun 8 12:52:20 2022) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False --== Host itdlolv101.ci.seattle.wa.us (id: 2) status ==-- conf_on_shared_storage : True Status up-to-date : True Hostname : itdlolv101.ci.seattle.wa.us Host ID : 2 Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Up"} Score : 3400 stopped : False Local maintenance : False crc32 : cc1c2261 local_conf_timestamp : 45453 Host timestamp : 45453 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=45453 (Wed Jun 8 12:55:15 2022) host-id=2 score=3400 vm_conf_refresh_time=45453 (Wed Jun 8 12:55:15 2022) conf_on_shared_storage=True maintenance=False state=GlobalMaintenance stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! [root@itdlolv101 ~]#

1 0

How to run virt-sysprep / How to troubleshoot template creation failure
by jeremy_tourville＠hotmail.com 08 Jun '22

08 Jun '22

When attempting to create a template the process fails. It has been suggested to run virt-sysprep manually and see why it it failed. Specifically, I have an Ubuntu 20.04 machine that doesn't boot properly even if the the template process does finish without error. I have tried creating the template several times as a test. about 80% of the time the template creation fails outright for qemu errors. The other 20% "appear" to work but the system boots to a grub emergency prompt Can someone clarify the process? I know you can run the command: virt-sysprep -d <name_of_vm> 1. Where do you run it from? The hypervisor host or the management engine? 2. What account do you need to use? What is the authentication username and password? I also think I read somewhere that you should refrain from using root. Anything else to know? Anything else to try regarding the grub prompt? Thanks.

2 5

Multiple displays
by mblecha＠flagshipbio.com 08 Jun '22

08 Jun '22

Our lab environment was set up on 4.4.10, then upgraded to 4.5, and our production environment was a clean install of 4.5. In our lab environment, I remember having to alter a setting back in 4.4.10 through 'engine-config' or other to get the multiple monitor option to cooperate in the VM settings. I googled around for the original article that allowed me to fix this on 4.4.10, but I cannot find what I had earlier. On the lab environment, the VM displays behave as expected; X starts and the second display becomes available in virt-viewer. In our production environment, same configurations, OS, etcetera on the VM, but X starts and the second display is still not available.

1 0

looking for some ISV backup software which integrates the backup API
by Nathanaël Blanchet 08 Jun '22

08 Jun '22

Hello, We are about to change our backup provider, and I find it is a great chance to choose a full supported ovirt backup solution. I currently use this python script vm-backup-scheduler (https://github.com/wefixit-AT/oVirtBackup) but it is not the workflow officially suggested by the community (https://www.ovirt.org/develop/release-management/features/storage/backup-re…) I've been looking for a long time an ISV who supports such an API, but the only one I found is this one : Acronis Backup Advanced suggested here https://access.redhat.com/ecosystem/search/#/ecosystem/Red%20Hat%20Enterpri… I ran the trial version, but it doesn't seem to do better than the vm-backup-scheduler script, and it doesn't seem to use the backup API (attach a clone as a disk to an existing vm). Can you suggest me some other ISV solutions, if they ever exist... or share me your backup experience?

5 4

nec-xhci: Win7 guest no response after ejecting usb storage
by lizhijian＠fujitsu.com 08 Jun '22

08 Jun '22

Hi folks, It's observed that win7 guest will become no response if i eject usb storage "Eject MiniKing" [cid:part1.8pN8Bh9e.qrkuS06h@fujitsu.com] the last screen after i clicked "Eject MingKing": [cid:part2.3Z0eywnJ.XOYVtZy1@fujitsu.com] After that, i have to hard reset the guest. Thanks Zhijian

3 4