[ovirt-users] Cannot activate storage domain

Sun Mar 11 16:49:05 UTC 2018

Shani,

  all storage domains are FC.

In the meantime I could track it down to corrupt vg metadata. I filed a bug (https://bugzilla.redhat.com/show_bug.cgi?id=1553133).

I ran vgcfgrestore, copied the VMs I could recover and recreated the LUNs. Would there have been another way of recovering?

All the best,
Simone

Von: Shani Leviim [mailto:sleviim at redhat.com]
Gesendet: Sonntag, 11. März 2018 14:09
An: Bruckner, Simone <simone.bruckner at fabasoft.com>
Cc: users at ovirt.org
Betreff: Re: [ovirt-users] Cannot activate storage domain

Hi Simone,
Sorry for the delay replying you.

Does the second storage domain you've mentioned is also an FC?
If so, please execute the following command  /usr/libexec/vdsm/fc-scan -v
on one host of each inactive storage domain and share the results you've got.

Regards,
Shani Leviim

On Thu, Mar 8, 2018 at 9:42 AM, Bruckner, Simone <simone.bruckner at fabasoft.com<mailto:simone.bruckner at fabasoft.com>> wrote:
Hi Shani,

  today I again lost access to a storage domain. So currently I have two storage domains that we cannot activate any more.

I uploaded the logfiles to our Cloud Service: [ZIP Archive]  logfiles.tar.gz<https://at.cloud.fabasoft.com/folio/public/1zf8q45o1e9l8334agryb2crdd>
I lost access today, March 8th 2018 around 0.55am CET
I tried to actived the storage domain around 6.40am CET

Please let me know if there is anything I can do to get this addressed.

Thank you very much,
Simone

________________________________
Von: users-bounces at ovirt.org<mailto:users-bounces at ovirt.org> [users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>]" im Auftrag von "Bruckner, Simone [simone.bruckner at fabasoft.com<mailto:simone.bruckner at fabasoft.com>]
Gesendet: Dienstag, 06. März 2018 10:19
An: Shani Leviim
Cc: users at ovirt.org<mailto:users at ovirt.org>
Betreff: Re: [ovirt-users] Cannot activate storage domain

Hi Shani,

  please find the logs attached.

Thank you,
Simone

Von: Shani Leviim [mailto:sleviim at redhat.com<mailto:sleviim at redhat.com>]
Gesendet: Dienstag, 6. März 2018 09:48
An: Bruckner, Simone <simone.bruckner at fabasoft.com<mailto:simone.bruckner at fabasoft.com>>
Cc: users at ovirt.org<mailto:users at ovirt.org>
Betreff: Re: [ovirt-users] Cannot activate storage domain

Hi Simone,
Can you please share your vdsm and engine logs?

Regards,
Shani Leviim

On Tue, Mar 6, 2018 at 7:34 AM, Bruckner, Simone <simone.bruckner at fabasoft.com<mailto:simone.bruckner at fabasoft.com><mailto:simone.bruckner at fabasoft.com<mailto:simone.bruckner at fabasoft.com>>> wrote:
Hello, I apologize for bringing this one up again, but does anybody know if there is a change to recover a storage domain, that cannot be activated?

Thank you,
Simone

Von: users-bounces at ovirt.org<mailto:users-bounces at ovirt.org><mailto:users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>> [mailto:users-bounces at ovirt.org<mailto:users-bounces at ovirt.org><mailto:users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>>] Im Auftrag von Bruckner, Simone
Gesendet: Freitag, 2. März 2018 17:03

An: users at ovirt.org<mailto:users at ovirt.org><mailto:users at ovirt.org<mailto:users at ovirt.org>>
Betreff: Re: [ovirt-users] Cannot activate storage domain

Hi all,

  I managed to get the inactive storage domain to maintenance by stopping all running VMs that were using it, but I am still not able to activate it.

Trying to activate results in the following events:

For each host:
VDSM <hostname> command GetVGInfoVDS failed: Volume Group does not exist: (u'vg_uuid: 813oRe-64r8-mloU-k9G2-LFsS-dXSG-hpN4kf',)

And finally:
VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'b83c159c-4ad6-4613-ba16-bab95ccd10c0',)

Is there anything I can do to recover this storage domain?

Thank you and all the best,
Simone

Von: users-bounces at ovirt.org<mailto:users-bounces at ovirt.org><mailto:users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>> [mailto:users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>] Im Auftrag von Bruckner, Simone
Gesendet: Donnerstag, 1. März 2018 17:57
An: users at ovirt.org<mailto:users at ovirt.org><mailto:users at ovirt.org<mailto:users at ovirt.org>>
Betreff: Re: [ovirt-users] Cannot activate storage domain

Hi,

  we are still struggling getting a storage domain online again. We tried to put the storage domain in maintenance mode, that led to “Failed to update OVF disks 809cc8d7-7687-46cf-a342-3be48674a9b3, OVF data isn't updated on those OVF stores”.

Trying again with ignoring OVF update failures put the storage domain in “preparing for maintenance”. We see the following message on all hosts: “Error releasing host id 26 for domain b83c159c-4ad6-4613-ba16-bab95ccd10c0 (monitor:578)”.

Querying the storage domain using vdsm-client on the SPM resulted in
# vdsm-client StorageDomain getInfo "storagedomainID"="b83c159c-4ad6-4613-ba16-bab95ccd10c0"
vdsm-client: Command StorageDomain.getInfo with args {'storagedomainID': 'b83c159c-4ad6-4613-ba16-bab95ccd10c0'} failed:
(code=358, message=Storage domain does not exist: (u'b83c159c-4ad6-4613-ba16-bab95ccd10c0',))

Any ideas?

Thank you and all the best,
Simone

Von: users-bounces at ovirt.org<mailto:users-bounces at ovirt.org><mailto:users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>> [mailto:users-bounces at ovirt.org<mailto:users-bounces at ovirt.org>] Im Auftrag von Bruckner, Simone
Gesendet: Mittwoch, 28. Februar 2018 15:52
An: users at ovirt.org<mailto:users at ovirt.org><mailto:users at ovirt.org<mailto:users at ovirt.org>>
Betreff: [ovirt-users] Cannot activate storage domain

Hi all,

  we run a small oVirt installation that we also use for automated testing (automatically creating, dropping vms).

We got an inactive FC storage domain that we cannot activate any more. We see several events at that time starting with:

VM perftest-c17 is down with error. Exit message: Unable to get volume size for domain b83c159c-4ad6-4613-ba16-bab95ccd10c0 volume 686376c1-4be1-44c3-89a3-0a8addc8fdf2.

Trying to activate the strorage domain results in the following alert event for each host:

VDSM <hostname> command GetVGInfoVDS failed: Volume Group does not exist: (u'vg_uuid: 813oRe-64r8-mloU-k9G2-LFsS-dXSG-hpN4kf',)

And after those messages from all hosts we get:

VDSM command ActivateStorageDomainVDS failed: Storage domain does not exist: (u'b83c159c-4ad6-4613-ba16-bab95ccd10c0',)
Failed to activate Storage Domain VMHOST_LUN_205 (Data Center Production) by <user>
Invalid status on Data Center Production. Setting status to Non Responsive.
Storage Pool Manager runs on Host vmhost003.fabagl.fabasoft.com<http://vmhost003.fabagl.fabasoft.com><http://vmhost003.fabagl.fabasoft.com> (Address: vmhost003.fabagl.fabasoft.com<http://vmhost003.fabagl.fabasoft.com><http://vmhost003.fabagl.fabasoft.com>), Data Center Production.

Checking the hosts with multipath –ll we see the LUN without errors.

We run oVirt 4.2.1 on CentOS 7.4. Hosts are CentOS 7.4 hosts with oVirt installed using oVirt engine.
Hosts are connected to about 30 FC LUNs (8 TB each) on two all-flash storage arrays.

Thank you,
Simone Bruckner

_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org><mailto:Users at ovirt.org<mailto:Users at ovirt.org>>
http://lists.ovirt.org/mailman/listinfo/users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180311/826734bb/attachment.html>