On Fri, Jun 7, 2019 at 5:03 PM <aminur.rahman@iongroup.com> wrote:
Hi
Has anyone experiencing the following issue with Storage Domain -

Failed to activate Storage Domain cLUN-R940-DC2-dstore01 --
VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'1b0ef853-fd71-45ea-8165-cc6047a267bc',)

Currently, the storge Domain is Inactive and strangely, the VMs are running as normal. We can't manage or extend the volume size of this storage domain. The pvscan shows as:
[root@uk1-ion-ovm-18  pvscan
  /dev/mapper/36000d310056978000000000000000014: Checksum error at offset 4397954425856
  Couldn't read volume group metadata from /dev/mapper/36000d310056978000000000000000014.
  Metadata location on /dev/mapper/36000d310056978000000000000000014 at 4397954425856 has invalid summary for VG.
  Failed to read metadata summary from /dev/mapper/36000d310056978000000000000000014
  Failed to scan VG from /dev/mapper/36000d310056978000000000000000014

This looks like corrupted vg metadata. 
I have tired the following steps:
1. Restarted ovirt-engine.service
2. tried to restore the metadata using vgcfgrestore but it failed with the following error:

[root@uk1-ion-ovm-19 backup]# vgcfgrestore 36000d310056978000000000000000014
  Volume group 36000d310056978000000000000000014 has active volume: .
  WARNING: Found 1 active volume(s) in volume group "36000d310056978000000000000000014".
  Restoring VG with active LVs, may cause mismatch with its metadata.
Do you really want to proceed with restore of volume group "36000d310056978000000000000000014", while 1 volume(s) are active? [y/n]: y

This is not safe, you cannot fix the VG while it is being used by oVirt.

You need to migrate the running VMs to other storage, or shut down the VMs. Then
deactivate this storage domain. Only then you can try to restore the VG.

  /dev/mapper/36000d310056978000000000000000014: Checksum error at offset 4397954425856
  Couldn't read volume group metadata from /dev/mapper/36000d310056978000000000000000014.
  Metadata location on /dev/mapper/36000d310056978000000000000000014 at 4397954425856 has invalid summary for VG.
  Failed to read metadata summary from /dev/mapper/36000d310056978000000000000000014
  Failed to scan VG from /dev/mapper/36000d310056978000000000000000014
  /etc/lvm/backup/36000d310056978000000000000000014: stat failed: No such file or directory

Looks like you don't have a backup in this host. You may have the most recent backup
on another host.
 
  Couldn't read volume group metadata from file.
  Failed to read VG 36000d310056978000000000000000014 from /etc/lvm/backup/36000d310056978000000000000000014
  Restore failed.

Please let me know if anyone knows any possible resolution.

David, we keep 2 metadata copies on the first PV. Can we use one of the copies on the PV
to restore the metadata to the least good state?

David, how do you suggest to proceed?

Nir