4:01 p.m.
Hi,
I experienced a weird Failure. I hope I explain it understandable and not too hard to
read
We had issues with our backups, that they can't finalize
Here is a Thread someone having the same issue.
https://forums.veeam.com/ovirt-kvm-f62/veeam-rhv-12-1-command-removeimage...
so I made some investigations and recognized, it seems to be an Issue with Snapshot
1. I created a Snapshot
https://paste.fo/c0b8e77a3400
what works
2. Then I tried to deleted the snapshot, what didn't work
https://paste.fo/013e5632e0d6
So I logged in to a node and checked several things
First strange thing was, I can see pvs vgs etc, I am used, that ovirt hides them in the
System (on one of those nodes it's still the case) but anyways
[root@ovnb05 ~]# pvs
WARNING: Couldn't find device with uuid uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb.
WARNING: VG 515bebca-972b-42ac-abff-d76af0071613 is missing PV
uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb (last written to
/dev/mapper/360002ac000000000000000880002a7fd).
PV VG Fmt
Attr PSize PFree
/dev/mapper/360002ac000000000000000870002a7fd 515bebca-972b-42ac-abff-d76af0071613 lvm2
a-- <12.00t 10.77t
/dev/sda3 rl lvm2
a-- 892.16g 0
[unknown] 515bebca-972b-42ac-abff-d76af0071613 lvm2
a-m <12.00t <12.00t
So I did
[root@ovnb05 ~]# pvdisplay
WARNING: Couldn't find device with uuid uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb.
WARNING: VG 515bebca-972b-42ac-abff-d76af0071613 is missing PV
uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb (last written to
/dev/mapper/360002ac000000000000000880002a7fd).
--- Physical volume ---
PV Name /dev/mapper/360002ac000000000000000870002a7fd
VG Name 515bebca-972b-42ac-abff-d76af0071613
PV Size 12.00 TiB / not usable 384.00 MiB
Allocatable yes
PE Size 128.00 MiB
Total PE 98301
Free PE 88247
Allocated PE 10054
PV UUID D6Typd-s7lA-PyoI-PvrL-rd9N-mS64-VeV1j3
--- Physical volume ---
PV Name [unknown]
VG Name 515bebca-972b-42ac-abff-d76af0071613
PV Size 12.00 TiB / not usable 384.00 MiB
Allocatable yes
PE Size 128.00 MiB
Total PE 98301
Free PE 98301
Allocated PE 0
PV UUID uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb
--- Physical volume ---
PV Name /dev/sda3
VG Name rl
PV Size 892.16 GiB / not usable 2.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 228393
Free PE 0
Allocated PE 228393
PV UUID kRe0EF-736L-QwZA-iGtM-67wp-8cr4-DKKcX2
I have never seen an unknown before, so I guess something happened to that LUN
But the LUN is available and in /var/log/messages I can't find any issue with the LUN
or the path etc.
[root@ovnb05 data-center]# blkid |grep uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb
/dev/sdi: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb"
TYPE="LVM2_member"
/dev/sdg: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb"
TYPE="LVM2_member"
/dev/sde: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb"
TYPE="LVM2_member"
/dev/sdn: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb"
TYPE="LVM2_member"
/dev/sdc: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb"
TYPE="LVM2_member"
/dev/sdl: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb"
TYPE="LVM2_member"
/dev/sdr: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb"
TYPE="LVM2_member"
/dev/sdp: UUID="uHJs0a-T7R3-2H8E-15W9-cGqW-ZN1e-xsdCsb"
TYPE="LVM2_member"
So I did some nasty stuff, like removing it and trying to add it again, through that
process it told me that there is no meta Data on the Device
So at the End I am a little bit lost now.
Maybe the storage was failing for some reason or maybe veeam messed something up, but what
leaves me afraid is, that the Ovirt-Engine didn't show me any Problem with the Cluster
or the Storage.
I mean it's obvious, that there was/is a huge failure with the storage and Ovirt did
not recognize neither did it alarm me.
At the moment I detached the LUN from the Storage Domain and created a seperate one (cause
I recognized that having several LUN's in a Storage Domain seems to be an bad Idea)
and one PVS is now hidden, the other one not
[root@ovnb05 data-center]# pvs
PV VG Fmt
Attr PSize PFree
/dev/mapper/360002ac000000000000000870002a7fd 515bebca-972b-42ac-abff-d76af0071613 lvm2
a-- <12.00t 11.81t
/dev/sda3 rl lvm2
a-- 892.16g 0
[root@ovnb05 data-center]# cat /etc/lvm/devices/system.devices
# LVM uses devices listed in this file.
# Created by LVM command vgs pid 6123 at Fri Feb 16 18:16:35 2024
VERSION=1.1.7
IDTYPE=mpath_uuid IDNAME=mpath-360002ac000000000000000870002a7fd
DEVNAME=/dev/mapper/360002ac000000000000000870002a7fd
PVID=D6Typds7lAPyoIPvrLrd9NmS64VeV1j3
IDTYPE=sys_wwid IDNAME=naa.61c721d06b5fb2002c49fbecd02f9d90 DEVNAME=/dev/sda3
PVID=kRe0EF736LQwZAiGtM67wp8cr4DKKcX2 PART=3
[root@ovnb05 data-center]# multipath -ll
360000970000197800382533030303031 dm-5 EMC,SYMMETRIX
size=5.6M features='1 queue_if_no_path' hwhandler='0' wp=ro
`-+- policy='service-time 0' prio=1 status=active
`- 9:0:25:0 sdj 8:144 active ready running
360002ac000000000000000870002a7fd dm-3 3PARdata,VV
size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 9:0:8:1 sdh 8:112 active ready running
|- 9:0:0:1 sdb 8:16 active ready running
|- 9:0:1:1 sdd 8:48 active ready running
|- 9:0:2:1 sdf 8:80 active ready running
|- 10:0:7:1 sdq 65:0 active ready running
|- 10:0:0:1 sdk 8:160 active ready running
|- 10:0:1:1 sdm 8:192 active ready running
`- 10:0:2:1 sdo 8:224 active ready running
360002ac000000000000000880002a7fd dm-4 3PARdata,VV
size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 9:0:8:2 sdi 8:128 active ready running
|- 9:0:0:2 sdc 8:32 active ready running
|- 9:0:1:2 sde 8:64 active ready running
|- 9:0:2:2 sdg 8:96 active ready running
|- 10:0:7:2 sdr 65:16 active ready running
|- 10:0:0:2 sdl 8:176 active ready running
|- 10:0:1:2 sdn 8:208 active ready running
`- 10:0:2:2 sdp 8:240 active ready running
360002ac0000000000000008f0002a7fd dm-26 3PARdata,VV
size=12T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 10:0:1:3 sdt 65:48 active ready running
|- 10:0:0:3 sds 65:32 active ready running
|- 10:0:7:3 sdv 65:80 active ready running
|- 10:0:2:3 sdu 65:64 active ready running
|- 9:0:1:3 sdx 65:112 active ready running
|- 9:0:2:3 sdy 65:128 active ready running
|- 9:0:0:3 sdw 65:96 active ready running
`- 9:0:8:3 sdz 65:144 active ready running
If I forgot something or you need more input just let me know, maybe you can light me up
what's wrong
thx in advance for any help/suggestions