1:10 p.m.
Hi,
last Sunday I experienced a power outage with one of my two oVirt
hypervisors. After power was restored I experienced some weirdness:
- on one of the VMs running on this hypervisor the boot disk changed, so
it was no longer able to boot. Looking at the console the VM would hang
on "Booting from hard disk". After I noticed that the wrong virtual disk
was marked as OS/bootable I got it booting again after correcting it to
the proper boot disk. This was done from the oVirt management server.
- on another VM I tried today to add another virtual disk to expand a
LVM volume. In dmesg I can see the new device:
[17167560.005768] vdc: unknown partition table
However, when I tried to run pvcreate I got an error message saying that
this was already marked as an LVM disk, and then running pvs give me the
following error:
# pvs
Couldn't find device with uuid 7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7.
PV VG Fmt Attr PSize PFree
/dev/vda5 rit-kvm-ssweb02 lvm2 a-- 59.76g 0
/dev/vdb1 vg_syncshare lvm2 a-- 500.00g 0
/dev/vdc1 VG_SYNCSHARE01 lvm2 a-- 400.00g 0
unknown device VG_SYNCSHARE01 lvm2 a-m 1024.00g 0
As you can see there's already a PV called /dev/vdc1, as well another
one named "unknown device". These two PVs belong to a VG that does NOT
belong to this VM, VG_SYNCSHARE01. The uuids for these two PVs are:
--- Physical volume ---
PV Name unknown device
VG Name VG_SYNCSHARE01
PV Size 1024.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 262143
Free PE 0
Allocated PE 262143
PV UUID 7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7
--- Physical volume ---
PV Name /dev/vdc1
VG Name VG_SYNCSHARE01
PV Size 400.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 102399
Free PE 0
Allocated PE 102399
PV UUID oKSDoo-3pxU-0uue-zQ0H-kv1N-lyPa-P2M2FY
The two PVs which doesn't belong on this VM actually belongs to a
totally different VM.
On VM number two:
# pvdisplay
--- Physical volume ---
PV Name /dev/vdb1
VG Name VG_SYNCSHARE01
PV Size 1024.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 262143
Free PE 0
Allocated PE 262143
PV UUID 7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7
--- Physical volume ---
PV Name /dev/vdd1
VG Name VG_SYNCSHARE01
PV Size 400.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 102399
Free PE 0
Allocated PE 102399
PV UUID oKSDoo-3pxU-0uue-zQ0H-kv1N-lyPa-P2M2FY
As you can see, same uuid and VG name, but two different VMs.
My setup:
oVirt manager: oVirt 3.5 running on CentOS 6.7
oVirt hypervisors: two oVirt 3.5 servers running on CentOS 6.7
During the time of the power outage mentioned earlier I was running
oVirt 3.4, but I upgraded today and rebooted the manager and both
hypervisors, but NOT the VMs.
Virtual machines:
Debian wheezy 7.9 x86_64
Storage:
HP LeftHand iSCSI
I have tried to locate error messages in the logs which can be related
to this behaviour, but so far no luck :(
--
Morten A. Middelthon
Email: morten(a)flipp.net