Snapshots and mess with Storage
by thorsten.bolten＠googlemail.com
Hi,
I experienced a weird Failure. I hope I explain it understandable and not too hard to read
We had issues with our backups, that they can't finalize
Here is a Thread someone having the same issue.
https://forums.veeam.com/ovirt-kvm-f62/veeam-rhv-12-1-command-removeimage...
so I made some investigations and recognized, it seems to be an Issue with Snapshot
1. I created a Snapshot
https://paste.fo/c0b8e77a3400
what works
2. Then I tried to deleted the snapshot, what didn't work
https://paste.fo/013e5632e0d6
So I logged in to a node and checked several things
First strange thing was, I can see pvs vgs etc, I am used, that ovirt hides them in the System (on one of those nodes it's still the case) but anyways
[root@ovnb05 ~]# pvs
WARNING: Couldn't find device with uuid uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb.
WARNING: VG 515bebca-972b-42ac-abff-d76af0071613 is missing PV uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb (last written to /dev/mapper/360002ac000000000000000880002a7fd).
PV VG Fmt Attr PSize PFree
/dev/mapper/360002ac000000000000000870002a7fd 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-- <12.00t 10.77t
/dev/sda3 rl lvm2 a-- 892.16g 0
[unknown] 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-m <12.00t <12.00t
So I did
[root@ovnb05 ~]# pvdisplay
WARNING: Couldn't find device with uuid uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb.
WARNING: VG 515bebca-972b-42ac-abff-d76af0071613 is missing PV uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb (last written to /dev/mapper/360002ac000000000000000880002a7fd).
--- Physical volume ---
PV Name /dev/mapper/360002ac000000000000000870002a7fd
VG Name 515bebca-972b-42ac-abff-d76af0071613
PV Size 12.00 TiB / not usable 384.00 MiB
Allocatable yes
PE Size 128.00 MiB
Total PE 98301
Free PE 88247
Allocated PE 10054
PV UUID D6Typd-s7lA-PyoI-PvrL-rd9N-mS64-VeV1j3
--- Physical volume ---
PV Name [unknown]
VG Name 515bebca-972b-42ac-abff-d76af0071613
PV Size 12.00 TiB / not usable 384.00 MiB
Allocatable yes
PE Size 128.00 MiB
Total PE 98301
Free PE 98301
Allocated PE 0
PV UUID uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb
--- Physical volume ---
PV Name /dev/sda3
VG Name rl
PV Size 892.16 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 228393
Free PE 0
Allocated PE 228393
PV UUID kRe0EF-736L-QwZA-iGtM-67wp-8cr4-DKKcX2
I have never seen an unknown before, so I guess something happened to that LUN
But the LUN is available and in /var/log/messages I can't find any issue with the LUN or the path etc.
[root@ovnb05 data-center]# blkid |grep uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb
/dev/sdi: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member"
/dev/sdg: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member"
/dev/sde: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member"
/dev/sdn: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member"
/dev/sdc: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member"
/dev/sdl: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member"
/dev/sdr: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member"
/dev/sdp: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb" TYPE="LVM2_member"
So I did some nasty stuff, like removing it and trying to add it again, through that process it told me that there is no meta Data on the Device
So at the End I am a little bit lost now.
Maybe the storage was failing for some reason or maybe veeam messed something up, but what leaves me afraid is, that the Ovirt-Engine didn't show me any Problem with the Cluster or the Storage.
I mean it's obvious, that there was/is a huge failure with the storage and Ovirt did not recognize neither did it alarm me.
At the moment I detached the LUN from the Storage Domain and created a seperate one (cause I recognized that having several LUN's in a Storage Domain seems to be an bad Idea) and one PVS is now hidden, the other one not
[root@ovnb05 data-center]# pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/360002ac000000000000000870002a7fd 515bebca-972b-42ac-abff-d76af0071613 lvm2 a-- <12.00t 11.81t
/dev/sda3 rl lvm2 a-- 892.16g 0
[root@ovnb05 data-center]# cat /etc/lvm/devices/system.devices
# LVM uses devices listed in this file.
# Created by LVM command vgs pid 6123 at Fri Feb 16 18:16:35 2024
VERSION=1.1.7
IDTYPE=mpath_uuid IDNAME=mpath-360002ac000000000000000870002a7fd DEVNAME=/dev/mapper/360002ac000000000000000870002a7fd PVID=D6Typds7lAPyoIPvrLrd9NmS64VeV1j3
IDTYPE=sys_wwid IDNAME=naa.61c721d06b5fb2002c49fbecd02f9d90 DEVNAME=/dev/sda3 PVID=kRe0EF736LQwZAiGtM67wp8cr4DKKcX2 PART=3
[root@ovnb05 data-center]# multipath -ll
360000970000197800382533030303031 dm-5 EMC,SYMMETRIX
size=5.6M features='1 queue_if_no_path' hwhandler='0' wp=ro
`-+- policy='service-time 0' prio=1 status=active
`- 9:0:25:0 sdj 8:144 active ready running
360002ac000000000000000870002a7fd dm-3 3PARdata,VV
size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 9:0:8:1 sdh 8:112 active ready running
|- 9:0:0:1 sdb 8:16 active ready running
|- 9:0:1:1 sdd 8:48 active ready running
|- 9:0:2:1 sdf 8:80 active ready running
|- 10:0:7:1 sdq 65:0 active ready running
|- 10:0:0:1 sdk 8:160 active ready running
|- 10:0:1:1 sdm 8:192 active ready running
`- 10:0:2:1 sdo 8:224 active ready running
360002ac000000000000000880002a7fd dm-4 3PARdata,VV
size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 9:0:8:2 sdi 8:128 active ready running
|- 9:0:0:2 sdc 8:32 active ready running
|- 9:0:1:2 sde 8:64 active ready running
|- 9:0:2:2 sdg 8:96 active ready running
|- 10:0:7:2 sdr 65:16 active ready running
|- 10:0:0:2 sdl 8:176 active ready running
|- 10:0:1:2 sdn 8:208 active ready running
`- 10:0:2:2 sdp 8:240 active ready running
360002ac0000000000000008f0002a7fd dm-26 3PARdata,VV
size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 10:0:1:3 sdt 65:48 active ready running
|- 10:0:0:3 sds 65:32 active ready running
|- 10:0:7:3 sdv 65:80 active ready running
|- 10:0:2:3 sdu 65:64 active ready running
|- 9:0:1:3 sdx 65:112 active ready running
|- 9:0:2:3 sdy 65:128 active ready running
|- 9:0:0:3 sdw 65:96 active ready running
`- 9:0:8:3 sdz 65:144 active ready running
If I forgot something or you need more input just let me know, maybe you can light me up what's wrong
thx in advance for any help/suggestions
2 months, 2 weeks
ovirt-engine-appliance bad gpg key
by Nathanaël Blanchet
Hello,
I tried to reinstall HE from a backup but I can't download ovirt-
engine-appliance because of a GPG failure:
2024-02-28 17:26:32,642+0000 ERROR
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:113 fatal: [localhost]: FAILED! =>
{"attempts": 10, "changed": false, "msg": "Failed to validate GPG
signature for ovirt-engine-appliance-4.5-20231201120252.1.el8.x86_64:
Public key for ovirt-engine-appliance-4.5-
20231201120252.1.el8.x86_64.rpm is not installed
Thank you to help me how to do because there is non -nogpgcheck option
as a workaround.
--
Nathanaël Blanchet
Administrateur Systèmes et Réseaux
Service Informatique et REseau (SIRE)
Département des systèmes d'information
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
blanchet(a)abes.fr
2 months, 3 weeks
virtio-scsi. SSD vs HDD
by duparchy＠esrf.fr
HI,
Is there a way to set a disk as being SSD (which really is). Or, better, set an entire Storage Domain being SSD ?
This would prevent the schedule optimization to be automatically ON in Windows. and maybe enable other underlying optimization (TRIM ?)
2 months, 3 weeks
VM actual size is showing more than the virtual size
by rootpentester20＠gmail.com
Hi team,
I am using Ovirt 4.4 and ISCSI as my storage domain. Some of my VMs have actual sizes showing more than virtual size and when we look inside the Vm the used space is much less than the actual size.
What could be an issue here and how can we resolve the same? Any help will be highly appreciated.
2 months, 3 weeks
hosted engine deployment fails with certificate error
by jerome lardeux
Hello
I,m trying to deploy a hosted engine in my lab, but i keep getting the same error
Every time, while trying to install evirt packages, i got this:
"ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost -> 192.168.222.91]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'ovirt-45-upstream': Cannot prepare internal mirrorlist: Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://mirrorlist.ovirt.org/mirrorlist-ovirt-4.5-el8 [SSL certificate problem: self signed certificate in certificate chain]", "rc": 1, "results": []}"
it is an ovirt node 4.5
thank you for your help
2 months, 3 weeks
Locked VHDD - Help To Unlock/Delete
by Matthew J Black
Hi All,
I've gone and made a (bad) mistake.
When creating a new VM I accidentally selected the wrong Storage Domain for the VM's VHDD; a Storage Domain with less space available than the size of the new (thick provisioned) VHDD.
So now I want to delete that VHDD (and possibly the accompanying VM) and start over.
However, the VHDD is "Locked" and I can't figure out a way to delete it (or Unlock it, or delete the VM, or Unattach the VHDD, etc, etc, etc). I can't find anything online so hopefully someone here can help me out.
For the record I have full Admin access to *all* physical and virtual resources and I don't mind doing things via CLI (which I suspect is what's going to be required). I'm also familia with SQL (MariaDB, MS-SQL) but not Postgres, so if this is going to required manipulating the oVirt DB then so-be-it, I can do that as well.
Thanks in advance
Cheers
Dulux-Oz
2 months, 3 weeks
ovirt-45-upstream GPG Key Error
by Matthew J Black
Hi All,
We just picked up a GPG Key error when running `dnf install ovirt-engine-appliance` in preparation of a fresh oVirt v4.5.5 install on RL v9.3:
~~~
oVirt upstream for CentOS Stream 9 - oVirt 4.5 79 kB/s | 1.6 kB 00:00
Importing GPG key 0x24901D0C:
Userid : "oVirt <infra(a)ovirt.org>"
Fingerprint: 3C98 E81D B93D EA6D 54DE 690E 44E4 75CB 2490 1D0C
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-oVirt-4.5
Is this ok [y/N]: y
Key imported successfully
Import of key(s) didn't help, wrong key(s)?
Public key for ovirt-engine-appliance-4.5-20231201120201.1.el9.x86_64.rpm is not installed. Failing package is: ovirt-engine-appliance-4.5-20231201120201.1.el9.x86_64
GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oVirt-4.5
~~~
An error? An issue with the repo definition? Can it safely be ignored (normally I'd say "No" but its from the oVirt Tam's own repo...)? Is the fingerprint above the correct one (and for that matter, where is the GPG Key's Fingerprint recorded on the oVirt Website so that we can check compliance ourselves)?
Anyway, thought I'd let people know (further details can be provide upon request)
Cheers
Dulux-Oz
2 months, 3 weeks
Using External CA
by Brent Saner
Hello all-
I am looking for a methodology to force oVirt to use certs/keys managed and deployed externally. These are automated by our various automation systems and are configured to match our security policies. Rogue and/or non-transparent and/or standalone CAs in our org do not comply with our security policy.
If I pre-deploy certificates and keys, does oVirt engine still need to be a CA?
Similarly, our encryption policy requires use of RSA 4096 or ED25519 for SSH. This, as I understand it, is also not compatible with the RSA 2048 generated/used by the engine's internal CA. Is SSHing to this new host using this key necessary, or can I externally enroll a new host into a cluster or - if not - use different SSH key via the engine (likely derived from the above/below-mentioned certs, which may be RSA 4096)?
Essentially, could I:
1.
Pre-provision certificates and keys in /etc/pki/ovirt-*/ with the appropriate filenames<https://www.ovirt.org/develop/release-management/features/infra/pki.html#...>
2.
Run engine-setup for the first time (using an answers file)?
Will engine-setup balk at the existence of those files, silently overwrite them, or use them?
Thanks in advance.
Brent Saner
SENIOR SYSTEMS ENGINEER
Follow us on LinkedIn!
brent.saner(a)netfire.com
855-696-3834 Ext. 110
www.netfire.com
2 months, 3 weeks
VM disk went un-allocated after storage migration
by rootpentester20＠gmail.com
Hi Team,
I am using Ovirt 4.4, I added new storage LUN and did a storage migration of one VM disk which had 400GB of data. AFter migration, the disk became un-allocated in windows and now it's asking to re-initialize, I am afraid that data has been lost. Is there a way we can recover it back?
Thanks
2 months, 3 weeks
