[ovirt-users] Strange disk behaviour within VMs

[Morten A. Middelthon]

Hi,

last Sunday I experienced a power outage with one of my two oVirt 
hypervisors. After power was restored I experienced some weirdness:

- on one of the VMs running on this hypervisor the boot disk changed, so 
it was no longer able to boot. Looking at the console the VM would hang 
on "Booting from hard disk". After I noticed that the wrong virtual disk 
was marked as OS/bootable I got it booting again after correcting it to 
the proper boot disk. This was done from the oVirt management server.

- on another VM I tried today to add another virtual disk to expand a 
LVM volume. In dmesg I can see the new device:
[17167560.005768]  vdc: unknown partition table
However, when I tried to run pvcreate I got an error message saying that 
this was already marked as an LVM disk, and then running pvs give me the 
following error:

# pvs
   Couldn't find device with uuid 7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7.
   PV             VG              Fmt  Attr PSize    PFree
   /dev/vda5      rit-kvm-ssweb02 lvm2 a--    59.76g    0
   /dev/vdb1      vg_syncshare    lvm2 a--   500.00g    0
   /dev/vdc1      VG_SYNCSHARE01  lvm2 a--   400.00g    0
   unknown device VG_SYNCSHARE01  lvm2 a-m  1024.00g    0

As you can see there's already a PV called /dev/vdc1, as well another 
one named "unknown device". These two PVs belong to a VG that does NOT 
belong to this VM, VG_SYNCSHARE01. The uuids for these two PVs are:

--- Physical volume ---
   PV Name               unknown device
   VG Name               VG_SYNCSHARE01
   PV Size               1024.00 GiB / not usable 3.00 MiB
   Allocatable           yes (but full)
   PE Size               4.00 MiB
   Total PE              262143
   Free PE               0
   Allocated PE          262143
   PV UUID               7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7

   --- Physical volume ---
   PV Name               /dev/vdc1
   VG Name               VG_SYNCSHARE01
   PV Size               400.00 GiB / not usable 3.00 MiB
   Allocatable           yes (but full)
   PE Size               4.00 MiB
   Total PE              102399
   Free PE               0
   Allocated PE          102399
   PV UUID               oKSDoo-3pxU-0uue-zQ0H-kv1N-lyPa-P2M2FY

The two PVs which doesn't belong on this VM actually belongs to a 
totally different VM.

On VM number two:

# pvdisplay
   --- Physical volume ---
   PV Name               /dev/vdb1
   VG Name               VG_SYNCSHARE01
   PV Size               1024.00 GiB / not usable 3.00 MiB
   Allocatable           yes (but full)
   PE Size               4.00 MiB
   Total PE              262143
   Free PE               0
   Allocated PE          262143
   PV UUID               7dDcyq-TZ6I-96Im-lfjL-cTUv-nff1-B11Mm7

   --- Physical volume ---
   PV Name               /dev/vdd1
   VG Name               VG_SYNCSHARE01
   PV Size               400.00 GiB / not usable 3.00 MiB
   Allocatable           yes (but full)
   PE Size               4.00 MiB
   Total PE              102399
   Free PE               0
   Allocated PE          102399
   PV UUID               oKSDoo-3pxU-0uue-zQ0H-kv1N-lyPa-P2M2FY

As you can see, same uuid and VG name, but two different VMs.

My setup:
oVirt manager: oVirt 3.5 running on CentOS 6.7
oVirt hypervisors: two oVirt 3.5 servers running on CentOS 6.7

During the time of the power outage mentioned earlier I was running 
oVirt 3.4, but I upgraded today and rebooted the manager and both 
hypervisors, but NOT the VMs.

Virtual machines:
Debian wheezy 7.9 x86_64

Storage:
HP LeftHand iSCSI

I have tried to locate error messages in the logs which can be related 
to this behaviour, but so far no luck :(

-- 
Morten A. Middelthon
Email: morten at flipp.net